Or just submit fixes as you find them.

References for building the docs:
http://www.postgresql.org/docs/current/static/docguide-toolsets.html
http://www.postgresql.org/docs/current/static/docguide-build.html

To build the PostgreSQL docs on Ubuntu, I had to install the following packages* additional to my base install:
- docbook
- docbook-utils (contains the collateindex.pl script)
- jade

Build the html docs from your top-level pg source dir:
./configure [options]
make html
…make is actually gmake on ubuntu.

This puts the html doc files in /path/to/pg_src/doc/src/sgml/html; view them with your favorite browser.

Make corrections on the sgml source files in /path/to/pg_src/doc/src/sgml; `make html` to re-generate the html docs.

I work with a git repo. Here’s how I create a patch:
git format-patch [id of version I want to patch]
The id is usually the version prior to my most recent commit.

*next time:
apt-get build-dep postgresql-doc
should pull down the packages needed to build the docs. Keep in mind this is going to pull the build requirements for the packaged version, which may not be the one you want (e.g., if the build requirements changed between versions; but this is unlikely).

Year 2 of OSBridge was almost as good as Year 1. I say almost, because I spent Monday (my first day off, completely to myself, in a loooong time) laid up with a nasty cold, and completely missed Tuesday, the first day of the conference. Bummer, because I was really looking forward to another installment of Hal Pomeranz’s Command Line Kung Fu . I’d caught this talk up at LFNW and it was truly excellent.

Wednesday

I arrived most of the way through The Rise of Hackerspaces (Leigh Honeywell), just in time to hear about chalkboard paint. How have I not known about this? More importantly, is there whiteboard paint? Oh yes, yes, there is. Leigh’s book recommendation: _Python for Software Design_: a good intro to programming concepts that just happens to use Python to illustrate.

Next, I participated in the Organizing User Groups panel. Eric Wilhelm briefly mentioned t-shirts, but I didn’t get the opportunity to thank him in public for ordering women’s versions of the pdx.pm t-shirts (twice!). Apparently there is a Drupal UG here in Portland, but they don’t advertise. We talked about ways to publicize groups, all-ages post-meeting activities, and presentation ideas. Sam Keen reports that having a few shorter talks instead of one long one is working well for PDXPHP.

The silliness of the compression algorithm and the overall entertainment value of Markus Roberts’ talk (Copyright Lawyers Can Goedel) both lived up to the high expectations I have for him.

chromatic’s Using Modern Perl made me want to try it out. You can

use Modern::Perl;

and have access to 5.10 features and (soon) 5.12 features, such as say, the ‘//’ operator, and named captures.

Best idea I got from this talk: put your module library in git, so if you upgrade something & it breaks, it’s super-easy to revert back to a working copy.

chromatic’s book is available on github, and he wants us to read it.

Schwern How to Report a Bug: I was disappointed that we didn’t get to see Schwern’s “REPENT! For the end of the UNIX Epoch is Nigh!” (I hear it’s showing at OSCON) but I will take this as a substitute. The basic premise is: it’s about empowering your users. There are a lot of gates (or filters, use your preferred terminology) in place to keep people from filing bugs. I ran into this a mere two days later, trying to file a bug report about pgadmin.

I skipped the last session so I could get dinner at the food carts & get back in time for Move Your Asana. We did this last year, and it’s a great way to do a quick decompression mid-conference.

Next up was the PostgreSQL BoF. Most of what we talked about was 9.0 – beta2 is coming out next week. (It actually came out June 4.)

Thursday

I had to work in the morning, and showed up for the afternoon sessions. I dropped in on Christof Pettus’s Introduction to PostgreSQL, because I’m planning to give a similar talk at LFNW next year. I have a pretty good idea what I want to cover, but figured I should make sure there wasn’t anything big I was missing. 45 minutes is only long enough for the briefest overview, but Christof covered it well, including some gotchas.

Next up, Jacinta Richardson’s Teach Your Class to Fish. This was geared more toward people giving multi-day intensive training sessions, but there’s good material that applies even to shorter one-day sessions. Like, be organized. I learned some new terms (eg “cognitive load”). Jacinta also stressed that it’s really important for students to take breaks – the kind where they actually leave the classroom & walk around. People who fall behind & try to catch up during lunch tend to fall even further behind. Put the most essential topics at the start of the day, because that’s when people are paying the most attention; save the easy topics for the end.

In You Shall Not Pass, Amye Scavarda and Chris Strahl talked about managing client expectations. I’m not a freelancer, but any lesson in boundary-setting is applicable to most work situations IMO. “If it’s not documented, it doesn’t exist” is a good rule of thumb to follow.

Paul “not a stalker; just a very nice man, who knows an awful lot about you” Fenwick taught us Practical Facebook Stalking. See the session notes & Paul’s blog for tips on tightening up your FB settings.

The Code-n-Splode BoF that night was basically a meet & greet and discussion of some things from the conference.

Friday

Voodoo donut truck! Woohoo! I ate too many donuts.

Went to Lance’s ShowOff presentation. I may try it…Mark Wong keep making me use Beamer for our presentations, which I guess is cool, but the layout gives me fits sometimes. I really like ShowOff’s syntax highlighting.

During the next session & into lunch (before I ran down to my favorite cart, the Whole Bowl), I worked on Maryanne’s Postgres install – got her logged in, and got her up to speed with the basic SQL to create & fill tables.

I’d intended to go home, but a couple of the after lunch sessions were just too juicy: Selena & Bart’s Advanced Trolling, followed by Audrey’s NSFW. You really Had to Be There ™ to fully experience these.

It is indeed very clicky, but it’s more of a light ticking sound than a deep thud like the M. You’ll wake up somebody in the next room, but not the neighbors. Unlike the M, it will not double as a personal defense device. But whooo boy, can I type fast, fast, fast on this thing. Sorta like driving a sports car for the first time, when the rear end cuts loose going around the corners.

Bonus: no markings on the keys means other people are afraid to use your computer.

Bottom line: I would definitely purchase one of these again. (But I sure hope it lasts over a decade like my M.)

So, say we have an interface that’s dropping packets. Not too many, but the ideal number is zero, so we’d like to see them in the graphs in our NMS (which is based on rrdtool as all decent NMSes are). We use rrdgraph to show packets out in black and packets that should have gone out, but were discarded instead in purple:

rrdtool graph images/router-errors-unscaled.png \
--title "unscaled" \
--vertical-label 'Pkts/Second' \
--start end-2day \
--end -1hr \
--width 800 \
--height 250 \
--imgformat PNG \
--interlace \
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE \
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE \
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE \
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+ \
LINE1:ifOutPkts#003300:ifOutPkts/sec \
LINE1:ifOutDiscards#990099:ifOutDiscards/sec\\n \
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\\n" \
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"

That produces a graph like this:

(Click on the thumbnails to get the full graphs.)

We can read the average rate of discarded packets in the graph key at the bottom, and there are tiny little blips in the purple line that represents discards , but we don’t have a strong visual cue that something is off.

One possible solution is to scale up the discard values relative to the total packets. A factor of 100 ought to do it. Then we’ll use the –right-axis options to rrdgraph to label the right-hand y-axis accordingly.

We add this CDEF to provide the scaling (the LINE1 etc commands will need to be altered accordingly; you’ll see those in the final snippet):

CDEF:scaled_ifOutDiscards=ifOutDiscards,100,*

That gives us a graph that looks like this:

Note that it now looks like we’re dropping up to 70 packets/second – we still have to read the stats in the key at the bottom of the graph. So let’s get the secondary y-axis correctly labeled & scaled, with the following commands:

--right-axis-label 'Discards/Second'
--right-axis 0.01:0

–right-axis-label prints the specified text along the right-hand axis.
–right-axis [scale:shift] scales and/or shifts the tickmarks on the right axis relative to the left axis. In this case, the new values we’re displaying are 100X the original values, so we need to scale our axis accordingly: 0.01. More simply: left/right = 1/100. We don’t need to start at a value other than 0, so we set the shift value to 0.

Hmmm…rrdtool has automatically converted our values to milli-units. (Note the lower-case m in the labels.) Let’s fix that with the –right-axis-format command:

--right-axis-format %1.1lf

And that’s all there is to it!

The final rrdgraph command looks like this:
rrdtool graph images/router-right-axis-format.png \
--title "right-axis-format" \
--vertical-label 'Pkts/Second' \
--right-axis-label 'Discards/Second' \
--right-axis 0.01:0 \
--right-axis-format %1.1lf \
--start end-2day \
--end -1hr \
--width 800 \
--height 250 \
--imgformat PNG \
--interlace \
DEF:ifOutUcastPkts=router.rrd:ifOutUcastPkts:AVERAGE \
DEF:ifOutNUcastPkts=router.rrd:ifOutNUcastPkts:AVERAGE \
DEF:ifOutDiscards=router.rrd:ifOutDiscards:AVERAGE \
CDEF:scaled_ifOutDiscards=ifOutDiscards,100,* \
CDEF:ifOutPkts=ifOutUcastPkts,ifOutNUcastPkts,+ \
LINE1:ifOutPkts#003300:ifOutPkts/sec \
LINE1:scaled_ifOutDiscards#990099:ifOutDiscards/sec\\n \
GPRINT:ifOutPkts:AVERAGE:"Avg ifOutPkts %1.2lf\\n" \
GPRINT:ifOutDiscards:MAX:"Max ifOutDiscards %1.2lf"

I have a table that contains hostnames, cards, and the slots on the host chassis those cards are installed in. All datatypes are varchars. The sorting problem arises because I have some cards that are actually installed on other cards; those are referenced by a slot/subslot designation.

Here’s some sample data:

testytest=# SELECT * FROM cards ORDER BY card_slot;
hostname | card_slot |   card_model
----------+-----------+----------------
zucchini | 0         | card
zucchini | 0/0       | daughter card
zucchini | 0/1       | daughter card
zucchini | 1         | card
zucchini | 1/0       | daughter card
zucchini | 1/1       | daughter card
zucchini | 1/15      | daughter card
zucchini | 1/2       | daughter card
zucchini | 17        | something else
zucchini | 18        | something else
zucchini | 2         | another card
zucchini | 3         | another card
zucchini | 4         | another card
(13 rows)

I’d like to see 1/15 come after 1/2, and 17 & 18 come after 2, 3, & 4. (Like I said, it’s a minor annoyance. But still an annoyance.) I need to be able to sort both pieces in numerical order. I went through some weird gymnastics writing functions to split out & return each piece, but each of my solutions introduced other problems. (These instant gratification projects so rarely are, eh.)

Before I wrapped things up for the day, I read the PostgreSQL docs about array functions & operators. That percolated around in my brain and a solution came to me while I was out hiking over the weekend:

First, I reCAST string_to_array & made it return a set of integers:
CREATE OR REPLACE function foo(varchar(15)) RETURNS integer[]
AS $$
SELECT CAST(string_to_array($1, '/') AS integer[])
$$
LANGUAGE SQL;

Since Pg includes < and > array functions, I expected to be able to sort by the array my function returned:

testytest=# SELECT hostname, card_slot, card_model, foo(card_slot) AS sort_value
FROM cards
ORDER BY sort_value;
hostname | card_slot |   card_model   | sort_value
----------+-----------+----------------+------------
zucchini | 0         | card           | {0}
zucchini | 0/0       | daughter card  | {0,0}
zucchini | 0/1       | daughter card  | {0,1}
zucchini | 1         | card           | {1}
zucchini | 1/0       | daughter card  | {1,0}
zucchini | 1/1       | daughter card  | {1,1}
zucchini | 1/2       | daughter card  | {1,2}
zucchini | 1/15      | daughter card  | {1,15}
zucchini | 2         | another card   | {2}
zucchini | 3         | another card   | {3}
zucchini | 4         | another card   | {4}
zucchini | 17        | something else | {17}
zucchini | 18        | something else | {18}
(13 rows)

Voila. Works as expected on my small (~2000 rows) data set. It also fails as expected when passed bad data (eg something with a text string).

First, we had a duplicate field name in the original select.  Not a problem if you’re just doing a select, but if you want to create a table (temp or otherwise) from the data, it won’t work.  So we replaced the first num_rows with rows_in_bytes.

Also, reading over this 5 months after the original attemp, I realize it’s a lot clearer if we don’t use table aliases in the outer SELECTs.

Then, we got some advice from Greg Smith that we shouldn’t do joins on pg_class.relname – this can screw you up if you have different schemas with identical table names.  You want to use oids (which I’d always thought was not desirable, but I’m assured it’s ok if you’re doing it with the system tables – you don’t want your application to depend on them, though. )  So, instead, we match pg_namespace.oid with pg_class.relnamespace.

Selena’s illustration of how this works:
SELECT relname, relkind FROM pg_class
JOIN pg_namespace ON pg_namespace.oid = pg_class.relnamespace WHERE relkind = 'r' AND pg_namespace.nspname = 'public';

The new & improved version of the query can be found on the Pg wiki.

I wanted to compare the new query against the old, so I created a couple of temp tables containing the results… and discovered we had a couple of data discrepancies:  a few of our tables were listed twice in the original query results, with different values for num_rows, only one of which was correct for the current schema:

portal=# SELECT tablename, rows_in_bytes, num_rows FROM index_experiment_1
WHERE tablename IN  ('detectorid_count','stations','test_agg')
ORDER BY 1;
tablename     | rows_in_bytes | num_rows
------------------+---------------+----------
detectorid_count | 0 bytes       |        0
detectorid_count | 631 bytes     |      631
stations         | 22 bytes      |       22
stations         | 350 bytes     |      350
test_agg         | 0 bytes       |        0
test_agg         | 1386 bytes    |     1386
(6 rows)

portal=# SELECT count(*) from detectorid_count;
count
——-
0
(1 row)

portal=# SELECT count(*) from stations;
count
——-
350
(1 row)

portal=# SELECT count(*) from test_agg ;
count
——-
0
(1 row)

It turns out we’d run into the exact problem that Greg had warned us about.  The additional rows were from identically-named tables in other namespaces.

Find your namespaces:
portal=# SELECT nspname from pg_namespace order by 1;
nspname
--------------------
information_schema
pg_catalog
pg_temp_1
pg_temp_2
pg_toast
pg_toast_temp_1
pg_toast_temp_2
public
selena
wendell
(10 rows)

Find your data:
portal=# SELECT count(*) from selena.detectorid_count ;
count
-------
631
(1 row)

portal=# SELECT count(*) from wendell.stations ;
count
——-
22
(1 row)

portal=# SELECT count(*) from selena.test_agg ;
count
——-
1386
(1 row)

Note that these match the additional data from our original query.

Thanks, Greg!

–
* No, I haven’t finished reading it yet…I don’t read during the summer, I ride my bike.

In Lists and Recursions and Trees, Oh My!, David Fetter gave us some example of old kludges to get row numbers out of Pg – “Not only is it slow, but it’s wrong” – but you may not notice that subtle wrongness in huge data sets.  This really illustrated the value of testing your data.

After lunch, I went to Josh Berkus’s 5 steps to PostgreSQL Performance Tuning.

He gave us some rules of thumb for figuring out how much RAM & CPU you need, but also recommends hiring a hardware geek to design your system for you – because vendors lie.   Try hardware out before you purchase it, or definitely test them within the warranty period.  And, here’s another use case for pg_proctab (other than my own amusement):  capacity planning.

Tip:  Don’t use autovacuum for data warehousing applications, or where you have large number of writes happening at once.  Manually vacuum those.

(An additional tip from me:  if you’re using linux, try increasing the default readahead buffer from 1024K to at least 1M for an ~80% performance improvement.  See our [in]famous file systems talk for the graphs to back this up.)

–

Thanks for another wonderful conference experience, PgPeeps!  See you again soon!

For my first tech conferences a few years ago, I only went to sessions that were meaningful for my job.  I’ve since had a much better time (and learned more) by choosing which sessions I’ll attend based on the following criteria, in this order:
1) topic interestingness
2) speaker interestingess
3) relevance to my job duties

(See Tips #1 and #2 in Skud’s recent Ten tips for tech conference attendees post.)

So, right out of the gate at PgWest, I’m in a python talk* – Adrian K’s (of LinuxFestNW fame) discussion on Dabo.  Dabo’s a python desktop framework;  I program primarily in Perl, and I’ve never touched a desktop app.  Adrian’s example project was a management system for a plant nursery, which I *do* understand, so I had a point of reference into the material (the methods & options used to track plants made sense to me).  I really wanted to talk to him more about this app, but never caught up with him.  (The hallway track felt kind of rushed for me this time.)  I got a good idea for form validation – if user tries to enter a blank value where one is not allowed, they get a pop-up immediately and the original text (if there was any) is put back in the field, forcing the user to accept the original input or enter something new before they can proceed to the next field.  This is a step up from giving the user the error message after they’ve submitted the form.

Next we were on to JD’s keynote, featuring the usual heckling of and by the podium.

Then Mark’s & my talk about pg_proctab, which ended with some live demos & some audience participation, the way I like it.

A bunch of us went to lunch at Honeyhole Sandwiches, where I tried the “Texas Tease” – BBQ chicken.  The sandwich was excellent.  I *highly* recommend the fries.

Scott Bailey’s Temporal Data talk was *packed*.  He talked about the “period” datatype, featured in both his own (Chronos) and Jeff Davis’s PgTemporal project.  You can do unions & intersects on time periods.  I am thinking this would be a useful datatype for searching large tables of log entries.

Based on Scott’s talk, I decided to go to Jeff’s “Not Just UNIQUE” talk, because he would be discussing this in a little more detail.  This meant I missed the session on backup & recovery.  (See comment above about more material than I can fit in my schedule.)

I spent the last session partly in the hackers’ lounge, working on some pg_proctab wrapper scripts with Mark.

Then it was off to the EDB-sponsored after-party, where I caught up with Lloyd Albin, who spoke at PDXPUG about a year ago.  He brought me up-to-date on the work he’s done on the project, including a twitter feed to let clients know of updates, which I think is really cool.

–

*Which I was late to, because we were installing the snacks in the Hackers’ Lounge (thanks, Mark!)

Slides are on github!  Your homework is to find them.   Here are some highlights.

There are over 16K modules on the CPAN as of 11 Oct 2009.  Wow.

Jeff’s recommended changes from the default cpan shell configuration:
- auto_commit 1
- prerequisites_policy follow
- build_requires_install_policy yes
- prefer_installer MB (Module::Build)
- change your make_install_make_command and mbuild_install_mbuild_command to include your sudo command.

–

Spend 10 minutes & give something back to CPAN every time you install a module:  Simply set up CPAN::Reporter!

Recommendations:
- make sure that you set cc_author to ‘no’.  (In the latest version, that’s the default.)
- set it to prompt you to edit/send the report if the tests fail.  This way you can judge if the failure is due to your own
boneheadedness.
- you can set the transport value to a file to run reports without sending them.
- for help:  perldoc CPAN::Reporter::Config

For automatic continuous testing, set up CPAN::Reporter::Smoker.  (Doesn’t actually install anything, justs runs tests.)

Recommendations:
- don’t run it as root;  you are the canary in the coal mine.
- create a dedicated user that has essentially no privs on your machine.
- run it on a separate Perl install (core modules only).
- this is a cool place to use that RAND option for prefer_installer in CPAN.

–

Another cool tip that I REALLY DIG because I have systems with multiple perls & users associated with them:
Set your shebang line to:

#!/usr/bin/env perl

Saturday:
9am:  Jeff Davis:  PostgreSQL, Extensible to the Nth Degree.  Jeff’s talks usually melt my brain, and I like that.
10:15:  Conference Keynote.
11:30am:  Mark Wong: pg_proctab.  Turns out I’m giving this talk with Mark, even though my name’s not on the schedule.  I should probably show up.
1:45pm:  Scott Bailey:  Temporal Data or Magnus Hagander:  Secure PostgreSQL Deployment.  There will be a coin toss.
3:00pm:  Kevin Kempter:  Backup and Recovery.  There’s always something else to learn about this topic.
4:00pm:  Bill Karwin:  Practical Full-text Search.

Sunday:
9:00am:  Aaron Sheldon:  XML Data Warehousing.
10:15am:  David Fetter:  Lists and Recursion and Trees (Oh, My!)  I want to learn about Windowing functions, new with 8.4
11:15am:  Matt Smiley:  Basic Query Tuning Primer.  Another topic I could stand to learn more about.
1:30pm:  Tossup between David Wheeler:  pgTAP Unit Testing Best Practices and Josh Berkus:  5 Steps to PostgreSQL Performance.  I’ll probably go to Berkus’s talk because Wheeler is a sport about repeating his talks for PDXPUG.

Other fun stuff:

The Hacker lounge will be open for two days of geekery:  7:30 am – 4:30pm Saturday, and 9-4 on Sunday.
EnterpriseDB has stepped up to provide entertainment after the Saturday sessions.
I haven’t heard if there are Lightning Talks, but I have a couple of ideas for one.  You should too.

See you there!