I have a table that contains hostnames, cards, and the slots on the host chassis those cards are installed in. All datatypes are varchars. The sorting problem arises because I have some cards that are actually installed on other cards; those are referenced by a slot/subslot designation.

Here’s some sample data:

testytest=# SELECT * FROM cards ORDER BY card_slot;
hostname | card_slot |   card_model
----------+-----------+----------------
zucchini | 0         | card
zucchini | 0/0       | daughter card
zucchini | 0/1       | daughter card
zucchini | 1         | card
zucchini | 1/0       | daughter card
zucchini | 1/1       | daughter card
zucchini | 1/15      | daughter card
zucchini | 1/2       | daughter card
zucchini | 17        | something else
zucchini | 18        | something else
zucchini | 2         | another card
zucchini | 3         | another card
zucchini | 4         | another card
(13 rows)

I’d like to see 1/15 come after 1/2, and 17 & 18 come after 2, 3, & 4. (Like I said, it’s a minor annoyance. But still an annoyance.) I need to be able to sort both pieces in numerical order. I went through some weird gymnastics writing functions to split out & return each piece, but each of my solutions introduced other problems. (These instant gratification projects so rarely are, eh.)

Before I wrapped things up for the day, I read the PostgreSQL docs about array functions & operators. That percolated around in my brain and a solution came to me while I was out hiking over the weekend:

First, I reCAST string_to_array & made it return a set of integers:
CREATE OR REPLACE function foo(varchar(15)) RETURNS integer[]
AS $$
SELECT CAST(string_to_array($1, '/') AS integer[])
$$
LANGUAGE SQL;

Since Pg includes < and > array functions, I expected to be able to sort by the array my function returned:

testytest=# SELECT hostname, card_slot, card_model, foo(card_slot) AS sort_value
FROM cards
ORDER BY sort_value;
hostname | card_slot |   card_model   | sort_value
----------+-----------+----------------+------------
zucchini | 0         | card           | {0}
zucchini | 0/0       | daughter card  | {0,0}
zucchini | 0/1       | daughter card  | {0,1}
zucchini | 1         | card           | {1}
zucchini | 1/0       | daughter card  | {1,0}
zucchini | 1/1       | daughter card  | {1,1}
zucchini | 1/2       | daughter card  | {1,2}
zucchini | 1/15      | daughter card  | {1,15}
zucchini | 2         | another card   | {2}
zucchini | 3         | another card   | {3}
zucchini | 4         | another card   | {4}
zucchini | 17        | something else | {17}
zucchini | 18        | something else | {18}
(13 rows)

Voila. Works as expected on my small (~2000 rows) data set. It also fails as expected when passed bad data (eg something with a text string).

First, we had a duplicate field name in the original select.  Not a problem if you’re just doing a select, but if you want to create a table (temp or otherwise) from the data, it won’t work.  So we replaced the first num_rows with rows_in_bytes.

Also, reading over this 5 months after the original attemp, I realize it’s a lot clearer if we don’t use table aliases in the outer SELECTs.

Then, we got some advice from Greg Smith that we shouldn’t do joins on pg_class.relname – this can screw you up if you have different schemas with identical table names.  You want to use oids (which I’d always thought was not desirable, but I’m assured it’s ok if you’re doing it with the system tables – you don’t want your application to depend on them, though. )  So, instead, we match pg_namespace.oid with pg_class.relnamespace.

Selena’s illustration of how this works:
SELECT relname, relkind FROM pg_class
JOIN pg_namespace ON pg_namespace.oid = pg_class.relnamespace WHERE relkind = 'r' AND pg_namespace.nspname = 'public';

The new & improved version of the query can be found on the Pg wiki.

I wanted to compare the new query against the old, so I created a couple of temp tables containing the results… and discovered we had a couple of data discrepancies:  a few of our tables were listed twice in the original query results, with different values for num_rows, only one of which was correct for the current schema:

portal=# SELECT tablename, rows_in_bytes, num_rows FROM index_experiment_1
WHERE tablename IN  ('detectorid_count','stations','test_agg')
ORDER BY 1;
tablename     | rows_in_bytes | num_rows
------------------+---------------+----------
detectorid_count | 0 bytes       |        0
detectorid_count | 631 bytes     |      631
stations         | 22 bytes      |       22
stations         | 350 bytes     |      350
test_agg         | 0 bytes       |        0
test_agg         | 1386 bytes    |     1386
(6 rows)

portal=# SELECT count(*) from detectorid_count;
count
——-
0
(1 row)

portal=# SELECT count(*) from stations;
count
——-
350
(1 row)

portal=# SELECT count(*) from test_agg ;
count
——-
0
(1 row)

It turns out we’d run into the exact problem that Greg had warned us about.  The additional rows were from identically-named tables in other namespaces.

Find your namespaces:
portal=# SELECT nspname from pg_namespace order by 1;
nspname
--------------------
information_schema
pg_catalog
pg_temp_1
pg_temp_2
pg_toast
pg_toast_temp_1
pg_toast_temp_2
public
selena
wendell
(10 rows)

Find your data:
portal=# SELECT count(*) from selena.detectorid_count ;
count
-------
631
(1 row)

portal=# SELECT count(*) from wendell.stations ;
count
——-
22
(1 row)

portal=# SELECT count(*) from selena.test_agg ;
count
——-
1386
(1 row)

Note that these match the additional data from our original query.

Thanks, Greg!

–
* No, I haven’t finished reading it yet…I don’t read during the summer, I ride my bike.

In Lists and Recursions and Trees, Oh My!, David Fetter gave us some example of old kludges to get row numbers out of Pg – “Not only is it slow, but it’s wrong” – but you may not notice that subtle wrongness in huge data sets.  This really illustrated the value of testing your data.

After lunch, I went to Josh Berkus’s 5 steps to PostgreSQL Performance Tuning.

He gave us some rules of thumb for figuring out how much RAM & CPU you need, but also recommends hiring a hardware geek to design your system for you – because vendors lie.   Try hardware out before you purchase it, or definitely test them within the warranty period.  And, here’s another use case for pg_proctab (other than my own amusement):  capacity planning.

Tip:  Don’t use autovacuum for data warehousing applications, or where you have large number of writes happening at once.  Manually vacuum those.

(An additional tip from me:  if you’re using linux, try increasing the default readahead buffer from 1024K to at least 1M for an ~80% performance improvement.  See our [in]famous file systems talk for the graphs to back this up.)

–

Thanks for another wonderful conference experience, PgPeeps!  See you again soon!

For my first tech conferences a few years ago, I only went to sessions that were meaningful for my job.  I’ve since had a much better time (and learned more) by choosing which sessions I’ll attend based on the following criteria, in this order:
1) topic interestingness
2) speaker interestingess
3) relevance to my job duties

(See Tips #1 and #2 in Skud’s recent Ten tips for tech conference attendees post.)

So, right out of the gate at PgWest, I’m in a python talk* – Adrian K’s (of LinuxFestNW fame) discussion on Dabo.  Dabo’s a python desktop framework;  I program primarily in Perl, and I’ve never touched a desktop app.  Adrian’s example project was a management system for a plant nursery, which I *do* understand, so I had a point of reference into the material (the methods & options used to track plants made sense to me).  I really wanted to talk to him more about this app, but never caught up with him.  (The hallway track felt kind of rushed for me this time.)  I got a good idea for form validation – if user tries to enter a blank value where one is not allowed, they get a pop-up immediately and the original text (if there was any) is put back in the field, forcing the user to accept the original input or enter something new before they can proceed to the next field.  This is a step up from giving the user the error message after they’ve submitted the form.

Next we were on to JD’s keynote, featuring the usual heckling of and by the podium.

Then Mark’s & my talk about pg_proctab, which ended with some live demos & some audience participation, the way I like it.

A bunch of us went to lunch at Honeyhole Sandwiches, where I tried the “Texas Tease” – BBQ chicken.  The sandwich was excellent.  I *highly* recommend the fries.

Scott Bailey’s Temporal Data talk was *packed*.  He talked about the “period” datatype, featured in both his own (Chronos) and Jeff Davis’s PgTemporal project.  You can do unions & intersects on time periods.  I am thinking this would be a useful datatype for searching large tables of log entries.

Based on Scott’s talk, I decided to go to Jeff’s “Not Just UNIQUE” talk, because he would be discussing this in a little more detail.  This meant I missed the session on backup & recovery.  (See comment above about more material than I can fit in my schedule.)

I spent the last session partly in the hackers’ lounge, working on some pg_proctab wrapper scripts with Mark.

Then it was off to the EDB-sponsored after-party, where I caught up with Lloyd Albin, who spoke at PDXPUG about a year ago.  He brought me up-to-date on the work he’s done on the project, including a twitter feed to let clients know of updates, which I think is really cool.

–

*Which I was late to, because we were installing the snacks in the Hackers’ Lounge (thanks, Mark!)

Slides are on github!  Your homework is to find them.   Here are some highlights.

There are over 16K modules on the CPAN as of 11 Oct 2009.  Wow.

Jeff’s recommended changes from the default cpan shell configuration:
- auto_commit 1
- prerequisites_policy follow
- build_requires_install_policy yes
- prefer_installer MB (Module::Build)
- change your make_install_make_command and mbuild_install_mbuild_command to include your sudo command.

–

Spend 10 minutes & give something back to CPAN every time you install a module:  Simply set up CPAN::Reporter!

Recommendations:
- make sure that you set cc_author to ‘no’.  (In the latest version, that’s the default.)
- set it to prompt you to edit/send the report if the tests fail.  This way you can judge if the failure is due to your own
boneheadedness.
- you can set the transport value to a file to run reports without sending them.
- for help:  perldoc CPAN::Reporter::Config

For automatic continuous testing, set up CPAN::Reporter::Smoker.  (Doesn’t actually install anything, justs runs tests.)

Recommendations:
- don’t run it as root;  you are the canary in the coal mine.
- create a dedicated user that has essentially no privs on your machine.
- run it on a separate Perl install (core modules only).
- this is a cool place to use that RAND option for prefer_installer in CPAN.

–

Another cool tip that I REALLY DIG because I have systems with multiple perls & users associated with them:
Set your shebang line to:

#!/usr/bin/env perl

Saturday:
9am:  Jeff Davis:  PostgreSQL, Extensible to the Nth Degree.  Jeff’s talks usually melt my brain, and I like that.
10:15:  Conference Keynote.
11:30am:  Mark Wong: pg_proctab.  Turns out I’m giving this talk with Mark, even though my name’s not on the schedule.  I should probably show up.
1:45pm:  Scott Bailey:  Temporal Data or Magnus Hagander:  Secure PostgreSQL Deployment.  There will be a coin toss.
3:00pm:  Kevin Kempter:  Backup and Recovery.  There’s always something else to learn about this topic.
4:00pm:  Bill Karwin:  Practical Full-text Search.

Sunday:
9:00am:  Aaron Sheldon:  XML Data Warehousing.
10:15am:  David Fetter:  Lists and Recursion and Trees (Oh, My!)  I want to learn about Windowing functions, new with 8.4
11:15am:  Matt Smiley:  Basic Query Tuning Primer.  Another topic I could stand to learn more about.
1:30pm:  Tossup between David Wheeler:  pgTAP Unit Testing Best Practices and Josh Berkus:  5 Steps to PostgreSQL Performance.  I’ll probably go to Berkus’s talk because Wheeler is a sport about repeating his talks for PDXPUG.

Other fun stuff:

The Hacker lounge will be open for two days of geekery:  7:30 am – 4:30pm Saturday, and 9-4 on Sunday.
EnterpriseDB has stepped up to provide entertainment after the Saturday sessions.
I haven’t heard if there are Lightning Talks, but I have a couple of ideas for one.  You should too.

See you there!

There’ll be three days of talks & tutorials plus a hackers’ lounge.   After-party plans are nebulous at this time, but we are researching options.  (Psst–pub crawl!)

Come join the fun!

http://pugs.postgresql.org/node/584

Duke proposed a PDX.pm sprint to work on 5.10.1:
“I think if everyone learns how to

a) get a copy of the perl git repo
b) keep it in sync
c) run the perl test suite, including running a single test at a time
d) submit a small documentation patch

then we will have a great start.”

This is perfect for a first sprint: “small, solve-able tasks” that will get everyone up & running, plus have the potential to actually be productive.

Your first code sprint with a new group of people is like the first day on the job…except nobody realizes that we’re each FNGs*. It can take some time to figure out how to work together.

It’s really great if people get the stuff that’s going to take time from the code sprint out of the way beforehand. For example, cloning the perl5 git repo (step a):

:::=>git clone git://perl5.git.perl.org/perl.git

(Took me 13 minutes.)

Duke suggested some advance reading as well – how to use the repo (cd perl; perldoc pod/perlrepository.pod) (you will need to install perl-doc if you don’t have it; it was not installed on ubuntu.)

–

At the sprint: I don’t have the necessary experience with git yet, so most of my time was spent learning that, specifically in regards to task b) Keeping your repo in sync.

git branch
shows me that I am on the blead branch

git remote show
just an interface into .git/config

git pull
convenient combination of fetch (from remote) + merge (into local stuff)

git pull origin
…by default pulls your current branch.
git pull
…would have done the same thing (depending on your .git/config!)

The blead branch is NOT what perl 5.10.1 is going to be released from, so that’s not what we’re going to work with.

Side trip:
Duke has an excellent graphic that he & jhelwig created that illustrates git as applied to the perl development process**. (I think I convinced them to put it into an actual graphics program & make it available…) One of the tips I got from the diagram was that tags are immutable. They should not move. If they move, UR DOIN IT WRONG.

perldelta.pod – manually-maintained (!!!) filed that lists the changes in all versions of Perl.

OK back to work:
How to check out maint 5.10: (remember, blead is beyond 5.10 & has changes we don’t need to be concerned with.)

git checkout -b
…is a combination of checkout and branch

git branch -a
…shows me all the branches I have locally.

I want to create a *local* tracking branch for 5.10 maint.

git branch [whatever I want to call it] [branch I want (or anything that can uniquely id a commit!)]
eg:
git branch maint-5.10 origin/maint-5.10

Tip:
git config --global color.ui auto
- git branch -a – local branch is white, currently checked-out is green, remotes are red.

THEN I need to actually check out the local tracking branch (ie switch to it)
git checkout [name I gave it]
eg:
git checkout maint-5.10

git checkout -b maint-5.10 origin/maint-5.10 is a shortcut for the above two steps.

–
c) run the perl test suite, including running a single test at a time

For reference:
:::--> uname -a
Linux princess 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC 2009 i686 GNU/Linux

First, you have to build (opts from Schwern, modified)
sh Configure -Ode \
-DDEBUGGING \
-Dprefix=/usr/local/perl/maint-5.10 \
-Dusedevel \
-Duseithreads \
-Dccflags='-I/usr/local/include -I/opt/local/include -I/sw/include' -Dldflags='-L/usr/local/lib -L/opt/local/lib -L/sw/lib' \
-Dlibpth='/usr/local/lib /opt/local/lib /sw/lib /usr/lib' \
-Uversiononly \
-Uinstallusrbinperl $@

- DEBUGGING builds the debugging symbols
- usedevel is required or configure will BITCHBITCHBITCH.
- useithreads enables one interpreter thread to do something I hope I don’t ever have to deal with. Or something.

Then:
TEST_JOBS=9; export TEST_JOBS #parallelizes your tests
make test
to compile perl and then run the tests. This takes a while. Grab a beer. In fact, grab a pitcher. Pay attention though, because a failing test here might be something interesting to patch.

make test TEST_FILES=[file] to run a single test.
e.g.:
make test TEST_FILES=t/pod/podselect.t
This turned out to be unreliable because it’s running through all the dependencies first. Bleah.

This is a better method:
cd t; ./perl TEST ./pod/podselect.t
Although Duke had problems with the debugger tests.

Remember TEST tries to cd to ./t, so you need to give your path relative to that.
(@Theory has more about this in his blog.)
(This is actually covered in perlhack, we just didn’t find it in time.)

–
Another tip:
get the name of your branch on your prompt. You need __git_ps1. Which is…somewhere halfway through the second pitcher.

Then add that var to your bash prompt whereever you’d like it:
$(__git_ps1 “(%s) “)
…the (%s ) is just my preferred formatting. The whole thing looks like this:
PS1=’\n\u@\h\w/\n$(__git_ps1 “(%s) “):::–> ‘

You can get some pretty colored output with this:
https://launchpad.net/~git-core/+archive/ppa

–
git shortlog:
shows you the commit messages, grouped by author. Useless if you’re looking for anything other than “a bunch of people did things.”

git shortlog -se:
number of commits, grouped by unique author + email.

git log --pretty=oneline maint-5.10..blead
…shows you the differences between two versions. In the perl source, there may be duplicates (different commit ID for same commit.)

–
Now let’s actually *do* something.
Stuff that’s holding up perl5.10.1 (I think)
Other stuff is in perltodo, but I’m not seeing anything that I’m actually up for given my skillset & current BAC.

I need to look for a doc to patch.

–
Annnnnd we decided to go to whiffies for pies.

More later! This is going to be a regular event.

–

* F’ing New Guy.
** Keep in mind that I am slightly more familiar with the Postgres release process & was expecting something similar.

Typically, I managed to miss the keynotes both days. There is something about conferences which makes me sleep through my alarm.

It was really hard choosing which talks to attend. The results of the coin toss:

Wednesday:
Tcl/Tk: Grandpa might be old, but he can still kick your ass! I went to this primarily because I use Expect so much. (Well, I use Expect.pm, but I remember my roots.) Webb gave a good intro to Tcl/Tk (”Tickle-Tea-Kay!”) despite some initial technical difficulties. I finally figured out the brackets vs braces variable expansion.

Then I gave my talk. Thankfully, Impress did not surprise me. I now have my unicorn badge.

Spindle, Mutilate, & Metaprogram: This was really cool, although it seemed similar to things that came out of the Perl community a few years back. I’d like to see a throwdown between Markus Roberts & Damian Conway.

Assholes are killing your project: I only managed about 20 minutes of this talk before I got too depressed & had to leave. Sorry, Donnie! We’ll talk about this later.

I spent some time in the hall track & then hit the yoga session. This was an excellent pick-me-up after a day of talking and brain-filling, and set me up for my BoF and then some time at the pub.

Thursday:
Arrived too late for chromatic’s Intro to Parrot so hung out in the speaker lounge and watched Andy and Irving’s run-through of Virtualize vs Containerize: Fight! I love the mashups.

Next up was Emma McGratten’s Ask Forgiveness not Permission, which had a lot of excellent reasons (financial & otherwise) for using open source, but not many tips on how to subversively bring it into your organization. I’m sure I know someone who could give a talk about that. :cough:

Lunch today was the excellent KOiFusionPDX Food Cart! They came to the conference site & provided excellent korean tacos. (Yeah, I know, sounds weird – but TRUST ME.)

Speaking of trust…Trust the Vote sounds like an excellent project. Unfortunately the question period started devolving into political discussion, and I didn’t want to just dive right in there and ask them why the hell they’re using MySQL instead of PostgreSQL.

Maria Webster got her unicorn badge for Faking it Till I Make It. Check out her blog to see what geeky women are up to.

bzr vs git smackdown with Selena & Emma. I’ve already made up my mind (git all the way!), but it’s good to listen to alternatives.

The Meditiation for Geeks session didn’t go too well for me, because I was so tired that any time I got close to The Zone, I almost fell over onto a fellow PostgreSQL Smurf. Still, the yoga & meditation sessions are a great way to unwind prior to the post-con socializing & I’d like to see more of this.

Pg took over the room & had our PostgreSQL BoF, which replaced the regular PDXPUG meeting.

Josh Berkus was riding a bicycle around town, which made me inordinately happy. I want to see if we can provide more bikes for attendees next year.

Friday: The Unconference rocked my socks:
1. Emma Jane’s “Playing with yourself” about Open Source documentation teams. I am even sadder that I missed WOSCON. This got me totally excited to contribute to docs. (Especially for certain Perl modules – but that’s a discussion for another post.) Highlights: the conference team is working on a style guide, and a library of personas (which isn’t public yet)

2. I signed up with DayOn, a local volunteer effort. This will be fantastic once we can get people trained in what’s actually reasonable to ask for.

3. I did a Network Management Basics talk (”FCAPS: What the hell?!?”) with Ua and Adam. We talked about the FCAPS model & where various tools we use fit. A very high percentage of them are rrdtool-based, so we talked about that a bit as well. Adam showed us his munin install. I keep trying to find other people in town who are as into Net Management as I am…I sort of feel like I need a 12-step program sometimes. On the way out, Ua proposed a Super-Sekrit project which we’ll start working on in September. (Excitement!)

—
I saw up-close what it took to put on this conference and I’d like to congratulate the organizers on their success! Great job, and can’t wait until next year!