Pythian Group

Somebody at $work asked me how I use Git to deploy stuff, probably working under the false hypothesis that wisdom is in any way, shape or form affiliated with yours truly. The fool…

Yet, it is true that I had my share of tinkering with Git, and that I might have gleaned from my experience. So I sent him a couple of links leading to more wisdom-certified sources. They are listed at the end of this blog entry; if you have more good articles/blog posts that you’d like to add to the pile, please don’t be shy and mention them in the comments.

But I can’t just stop there, really. So, if you allow me, I’ll dust off the good ol’ soap-box, hop on it, and share my thoughts on the subject. As usual, don’t forget to take an adequatly-sized grain of salt to go with your popcorn.

Read the rest of this entry . . .

For my backups at home I have a DNS-323 which, after a minimal twist of the arm was applied, can be accessed via ‘ssh’ and ‘rsync’. Methodology-wise, I’m following the general idea described in this forum. Namely: for each of my machines I have one backup directory per week of the year (01, 02, 03, etc) and — and this is the über-cool part of it — rsync‘s ‘--link-dest‘ is used to hard-link files that didn’t change since the previous backup run. This means that I can have weekly snapshots of all my machines at the fraction of the space a full backup would take. Very nice.

But, just like gas in a vacuum, files have a nasty habit to fill up all available disk space. In my case, the 100% mark was hit last week. Obviously, I had to delete stuff. But… which stuff?

Sure, I have access to du on the DNS-323 to find out. But du’ing 350G worth of disk space is no fun. And there is the matter of the hard-links that further complicates things. Even though the backup directory ‘enkidu/03′ weights 5G, if 95% of its content is hard-linked by other weeks, deleting it won’t give me back much. To properly get the job done, what I really want is a way to get a snapshot of the disk usage locally, and in a format that would allow me to navigate and examine it at will.

Read the rest of this entry . . .

Dist::Zilla is a little bit of a two-edged blade. Whereas it enormously simplify things for the module author, it can also create quite the speed bump for the casual contributor. Not only the content of the repository can diverge strongly from the distribution (“dude, where’s the Build.PL? And what the heck happened to the POD?”), but to transmute that repository code into its distribution-ready incarnation, one has no choice to do the zilla dance. That is, if one can. In my experience it’s not unusual to stumble on distribution with a dist.ini that has plugins that are either onerous to get working on my machine or, worse, are not compatible with their latest versions on CPAN. Indeed, I can remember a few cases were it took more time to get dzil build to work than it took to write the code patch itself.

In an effort to remove that potential hurdle from my own distributions, I’ve developed a certain pattern that, so far, seems to do the trick. In my dist.ini, I leverage the wonderful Dist::Zilla::Plugin::Git and have:

[Git::Check]
[Git::Commit]

[Git::CommitBuild]
    release_branch = releases
[Git::Tag]
    tag_format = v%v
    branch     = releases

[Git::Push]
    push_to = github

The magic is in Git::CommitBuild. By itself, Git::CommitBuild commits the result of a dzil build on a branch called build/original branch. Those are throwaway branches that I typically never push to my public repository, but that are very useful to have a quick peek at what the Dist::Zilla-generated code will look like.

However, it’s the optional release_branch parameter that interest us here. When defined, the plugin also commits the build content to the given branch on a dzil release. In other words, it provides us with a branch containing the exact code that goes out on CPAN. It’s a tidy branch too, with one commit per release, just like what GitPAN or git-backpan-import from Git::CPAN::Patch would give you. Except that it’s even a bit better as it also keeps the relationship with the development branch it comes from. For example, here is what Catalyst::Plugin::MemoryUsage‘s repo looks like:

So, basically, the repository ends up having two main branches, master, on which the development is done, and releases, that automatically keeps track of the “real” released code.

GitHub can even sweeten the situation a little bit more. In the administration page of a project, it’s possible to switch the default branch from master to releases. That way, the casual downloaders/patchers/curious souls initially see a version of the code that they can relate with what’s on CPAN. And if they are brave enough to dive in its Dist::Zilla counterpart, it’s also there for them.

And there is a last, very nice bonus feature of that way of doing things. We can auto-populate the README.mkd of the project by using Dist::Zilla::Plugin::ReadmeMarkdownFromPod and adding the following to dist.ini:

[ReadmeMarkdownFromPod]

For an example of the resulting generated README.mkd, you have have a gander at my Task-BeLike-YANICK GitHub page.

Tell me if that sounds familiar: you’re happily hacking on your codebase and, at some point, you type in a password / secret token / really shameful limerick that shouldn’t be sent to the repository mothership, but that you need on your local copy. Well, no sweat, you just have to remember not to commit that specific file. So hack, hack, hack go the fingers. Several hours later, satisfied by your work, you commit the fruit of your labor and send it to the master public repository. And, guess what? The file you were supposed to remember not to commit? You didn’t. And you did. Ooops.

Read the rest of this entry . . .

I already knew about git-achievements (and partake in the fun). But earlier this week Dean showed me Unit Testing Achievements.

My first thought after seeing it was: “cool, I want to port that to Smolder!”. My second thought was: “hey, if we can do it for Git, and for testing, why not for Perl itself?” Perl::Critic and Perl::Tidy already use PPI for their magic, surely it wouldn’t be too hard to harness its power for a little bit of fun?

Read the rest of this entry . . .

[yanick@enkidu shuck]$ perl -MGit::Wrapper \
    -E'say for Git::Wrapper->new(".")->show( "master:shuck-and-awe-9" )'

First this week we have John Anderson filling us up on the Perl high drama of OSCON of earlier this week. In a nutshell the organizers provided, as it’s the tradition, ribbons to the attendees, and the Perl Mongers in the crowd got one reading Desperate Perl Hacker. The epithet, coined in an XML article written in 1997, was meant in good fun, but was received with a distinct lack of glee by the Perl hackers. Which is no surprise, considering how mongers already have to fight tooth and nail to dispel the heavy baggage of preconceptions that our language accumulated throughout the years. Quite a few blog entries sprouted to discuss the whole hooplah, with Piers Cawley‘s being one of the most eloquent, explaining that, yes Virginia, Perl can, and is, often used at the 11th hour to save someone’s bacon and, consequently, has instances of code that are less than perfectly pretty. But, it’s also much more than an emergency fire extinguisher and and has flip side that consists of a strong, solid and modern ecosystem that can, and is, definitively used for bigger projects.

Buuut enough about that. Let’s return to our regular parade of technological goodies, shall we?

Read the rest of this entry . . .

CC BY-NC-SA 2.0

There were two things I have wanted to do for some time now. The first was to come up with a way to quickly and easily set up a DarkPAN mirror so that we would have more control over our dependency chain at work. The second was to make a portable CPAN proxy service, so that I can always have access to my favorite modules, even if the machine I’m working on has no Internet access. Last week, I finally had a few ‘rount tuits’ to spend on this type of background itch, and the result is dpanneur (for dépanneur, French Canadian for convenience store).

Installation

As it stands, dpanneur is a very thin Catalyst application gluing together the goodiness of CPAN::Cache and MyCPAN::App::DPAN, and throwing in Git as the archive manager.

To get it running, first fetch it from Github

$ git clone git://github.com/yanick/dpanneur.git

then check that you have all the dependencies

$ perl Makefile.PL

and run the script that will create the module repository

$ ./script/create_repo

For now, the module repository is hard-coded to be in the subdirectory cpan of dpanneur. A branch called proxy is created and checked out. Eventually, I’ll use GitStore to push newly fetched modules to the repository, but for the time being if dpanneur is to be used as a proxy, that branch must remain as the one being checked out.

All that is left is to fire up the server in whichever mode you prefer (single-thread test server would do nicely for now)

$ ./script/dpanneur_server.pl

and there you are, running your first dpanneur. Congrats! :-)

Using it as a caching proxy

You can use the server as a caching proxy, either for its own sake, or to seed the DarkPAN branches. To do that, you just have to configure your CPAN client to use http://yourmachine:3000/proxy:

$ cpan
cpan[1]> o conf urllist = http://localhost:3000/proxy
cpan[2]> reload index
cpan[3]> install Acme::EyeDrops
Running install for module 'Acme::EyeDrops'
Running make for A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
Fetching with LWP:

http://localhost:3000/proxy/authors/id/A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz

etc..

As the modules are downloaded, they are also saved and committed within the repo

[dpanneur]$ cd cpan

[cpan (proxy)]$ git log -n 3
commit d065ad152f2204295334c5475104a3da517b6ae1
Author: Yanick Champoux <yanick@babyl.dyndns.org>
Date:   Wed Mar 10 20:32:52 2010 -0500

    authors/id/A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz

commit e8d2e83d1b16e2e0713d125f9a4bd2742681f859
Author: Yanick Champoux <yanick@babyl.dyndns.org>

Date:   Wed Mar 10 20:31:42 2010 -0500

    authors/id/D/DC/DCONWAY/Acme-Bleach-1.12.tar.gz

commit 7e0b4b600bac8424c519199ee96dc56ffbb177eb
Author: Yanick Champoux <yanick@babyl.dyndns.org>
Date:   Wed Mar 10 20:30:47 2010 -0500

    modules/03modlist.data.gz

Using it as a DarkPAN server

There is not much more involved to enabling DarkPAN repos. All we have to do is to create a branch with the modules we want and have the ‘dpan’ utility bundled with MyCPAN::App::DPAN generate the right files for us.

To continue with the example of the previous section, let’s say that we want a DarkPAN branch containing Acme::EyeDrops, but not Acme::Bleach. Then we’d do

                        # only necessary if you are running
                        # the server while you work on the branch
[dpanneur]$ git clone cpan cpan-work   

[dpanneur]$ cd cpan-work

                        # branch just before we imported Acme::Bleach
[cpan-work (proxy)]$ git branch pictoral 7e0b4b600bac8424c519199ee96dc56ffbb177eb

[cpan-work (proxy)]$ git checkout pictoral
Switched to branch 'pictoral'

                        # cherry-pick the Acme::EyeDrops commit
[cpan-work (pictoral)]$ git cherry-pick d065ad152f2204295334c5475104a3da517b6ae1

                        # rebuild the module list
[cpan-work (pictoral)]$ dpan

                        # commit the new 02packages.details.txt.gz
[cpan-work (pictoral)]$ git add .
[cpan-work (pictoral)]$ git commit -m "dpan processing"

                        # push back to the mothership
[cpan-work (pictoral)]$ git push origin pictoral

And that’s it. Now point the cpan client to http://yourmachine:3000/pictoral, and you’ll get the limited mirror.

cpan[1]> o conf urllist http://localhost:3000/pictoral
cpan[2]> reload index

cpan[3]> i Acme::EyeDrops
Strange distribution name [Acme::EyeDrops]
Module id = Acme::EyeDrops
    CPAN_USERID  ASAVIGE (Andrew J. Savige <asavige@cpan.org>)
    CPAN_VERSION 1.55
    CPAN_FILE    A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
    UPLOAD_DATE  2008-12-02
    MANPAGE      Acme::EyeDrops - Visual Programming in Perl
    INST_FILE    /usr/local/share/perl/5.10.0/Acme/EyeDrops.pm
    INST_VERSION 1.55

cpan[4]> i Acme::Bleach
Strange distribution name [Acme::Bleach]
No objects found of any type for argument Acme::Bleach

So mothers keep your hackers at home
Don’t let them journey all alone
Tell them this world is full of danger
And to shun the repositories of strangers
– The Tag Set of Strangers,
(with apologies to) Nick Cave and the Bad Seeds

One of the things I love about Git is how I can add branches from remote repositories in mine at will without fearing messing up anything. The remote branches will not clash with mine, even if they share the same names, because they are referenced as repository/branch. However, as for anything else, you can still poke yourself in the eye if you try hard enough:

Read the rest of this entry . . .

picture by Geophaps
Hey, that one in the sixth row…
Doesn’t he looks familiar?

So there I am, on my morning bus ride, reading my copy of The Definitive Guide to Catalyst (keep your eyes peeled for the upcoming review of the book in the Perl Review).

I’m near the end, in Chapter 11, Catalyst Cookbook. As it is with most tech books, the last chapters are the most engrossing, as the gloves finally come of and the writers throw at you all the wonderful, mind-bending stuff that the rest of the book prepares you for.

The section I’m at is about the development process. Specifically, it shows how you can put hooks in your versioning system to automatically screen commits to conform to Perl::Critic and Perl::Tidy policies. The given example script uses Git, which is just dandy with me as it is my current VCS of choice. But there’s something . . .  funny about that script. The way the utility functions are stashed at the end after a

### utility functions ##############################

line. The choice of variable names. The comments. It all feels oddly familiar. Read the rest of this entry . . .