A stats tracking application I’ve slowly built over the last six months or so recently started showing some potentially anomalous data for one of its views. A number that has trended upward steadily for all time suddenly dipped substantially (5%) and wallowed about at the low number for a week before beginning to trend upward again. During the low period, I looked into the matter to try to determine whether there was a problem with the stats or whether the numbers were legitimately low.

My first step was to identify any changes that had been made to the stats program immediately prior to the dip. There were a couple of changes that couldn’t possibly have related to the issue and one that seemed unlikely to have caused a problem but that merited investigation. So in a copy of the code and on a copy of the production database, I rolled back the possible culprit and re-crunched the stats. The numbers came back the same, so the possible culprit I had identified was in fact not the culprit.

Next, I wanted to determine whether or not I had made any recent changes to the database structure that could possibly have skewed my calculations (I do all sorts of calculations and caching on raw data). This wasn’t as straightforward as just viewing an svn log to determine when any relevant files were changed. But there is a way to tell when (if not how) a database was last changed. You need only look at the timestamps on the files that contain information about your database structure. This works for mysql, in any case. Here are the steps I took:

1. Figure out where your data files are stored. If you don’t know, try typing “ps -eaf |grep mysql” (or the equivalent on your linux distro) and looking for the “datadir” parameter in the command line string. Change into that directory.
2. There, you should see a directory whose name corresponds to the database you want to check for changes. Change into that directory.
3. Next, do something like “ls -la *.frm” or, if there are many tables, “ls -la <tablename>.frm”. This lists timestamps for the files that store table structure, which tells you when the tables were changed.

In my case, I have a production database and a staging database that stay pretty close to in sync. Luckily, I had synced data between the two but hadn’t synced a change I had made to a production table’s structure. So the timestamp for my production table corresponded to the date on which our stats dip appeared. I could then get a mysql prompt and type “show create table <tablename>” for both my production and my staging databases to see what the difference between them was.

It turned out to be that I had simply dropped an extraneous index in production, something that shouldn’t have affected my stats. That crunching stats in both the staging and the production environments had the same end result corroborates my conclusion that the index change didn’t matter.

As my stats had begun trending upward again and we had just come off the Thanksgiving holiday break, I felt pretty confident after doing this little diligence pronouncing the dip a product of holiday lag. In any case, now you know, if you didn’t already, how you can check to see when you last changed a mysql table’s structure.

powered by performancing firefox

It’s confirmed — Knoxville’s second Flock meetup will take place at 7:00 p.m. on Wednesday (tomorrow) at the Barne’s and Noble on Kingston Pike (yes, we do know about books here in the south). We’ll gather in the cafe area. We’re a small group so far (but bigger than last time), and anybody in the area is welcome to stop by. I’ve got some nifty buttons to give out and, FedEx willing, I’ll have one or two tee-shirts that people can fight over. Other than that, we’ll just talk Flock. I hope to have a chance to give an overview of things to come and to answer any questions I’m able to about where we are with the software. Admittedly, since I’m on the web end of things now rather than the client side, my knowledge on that front is more limited than in the past. In any case, it should be a good event. Naturally, unless it’s a real snoozer, I’ll report on how it went. Consider this an invitation to other community members to hold meetups and to do status reports afterward. You’ll have to check this with Community Ambassador Will Pate, but I gather we’re getting much closer now to being ready for spread-like campaigns, and meetups seem to me like as good a way as any to participate.

Back in March, I more or less presided over a small Flock meetup in Knoxville. We’ve come a long way since March, and given the recent releases, I thought it might be a good time to hold another meetup event, this time with hopefully a slightly broader reach. Including myself, I can count on four participants this time and may be able to garner a fifth. If I break six, I’ll be pretty happy; Knoxville isn’t exactly browseropolis, you know. If you happen to be in the Knoxville area and are interested in meeting some other Flock users or just want to find out more, please let me know by email (daryl at flock dot com), and I’ll fill you in on the details as I firm up plans. Tentatively, I’m looking at finding a book store or coffee shop with wifi on Wednesday or Thursday evening next week.

There’s no set agenda, but I imagine we’ll talk some about where the browser’s been, where it’s come, and where it’s going. I fully anticipate the airing of some beefs with the browser, and I hope we’ve also given reason for some kudos to be awarded as well.

So, people who know me know I work for Flock, which has something to do with computers. We actually produce a web browser built on the platform that Firefox is built on. When we first started talking big a little less than a year ago, we sang out about how we were going to revolutionize browsing. Talking big got us in a little trouble because we were over-hyped too early. We’ve since changed our message a bit. Rather than being the saviors of the internets, we’re just trying to make the web a little easier to use for some people. If you like to blog, share your bookmarks, and use photo services like flickr and photobucket, maybe Flock is something you’d be interested in. Otherwise, it’s probably not. Here’s some of what I personally like about our product; these things are what would make me think about using Flock if I didn’t feel bound to because I’m an employee.

Built-in news/feeds integration. Most blogs and many news sites you see nowadays display little orange radio-wave icons and advertise things like “RSS2.0″ and “Atom.” These are just techie ways of saying “here’s a list of recent articles that your software may be able to keep track of.” The idea is that you can plug these feeds into your feed reading software, which will notify you of updates so that you don’t have to go out each day and click through to the dozen sites you like to keep up with. There have long been Firefox extensions and standalone applications that would consume these feeds, but I’ve never liked any of them. In the case of the extensions, the interface has always bugged me. In the case of applications, well, I really don’t need another application open on my desktop. Flock has feed-reading built in. I subscribe to the feeds for the blogs I like to read. When one of them has new content, a button in my browser lights up to let me know. So no more going out and visiting the site to see what’s new. When I click the news button, a sidebar opens up that gives me a tree view of my feeds (sort of like an email view, with folders and items on the side), which I can scan easily for updated content. This feature really makes my life easier and is probably my favorite thing in the browser.

Next, there’s the star button. It’s a little button connected to the urlbar. If you’re at a site that you like and would like to save for later, just click the star button. It’ll turn orange, and when you later visit sites you’ve starred, it’ll turn orange to let you know you’ve already starred them. If you’re adventurous, you can make an advanced options box appear that’ll let you apply tags (just think of them as ad hoc labels that are easier to use than nested folders) to the things you star. We call the things you star “favorites.” And there’s a favorites manager that lets you see a list view of the things you’ve starred. You can search through them by tag, sort by recent, organize them into collections (which you can make appear in your toolbar), and go back to their advanced options to change settings. What’s more, if you’re so inclined, you can set your favorites to be synced up with an online bookmarks service like delicious or Shadows so that your favorites are available to you anywhere, any time. I like our favorites system because it lets me do some level of organization of my links without making me manipulate a bunch of nested folders; I can quickly apply multiple labels that let me get to my links later in several ways without having to file them redundantly in multiple folders.

Favorites interact with the search box, which widget can be very useful but is something I often overlook. If you’ve used Firefox before, you’re already familiar with the search box embedded in the browser interface. You can select from among several search engines to direct your searches to, and you can easily perform searches without first having to navigate to the search page itself. I have muscle memory that makes me hit my home button and search straight from Google, so the search box hasn’t historically been something I’ve used (hence my often overlooking it now). But there’s one thing that makes it very handy in Flock: As you type search strings, it searches through your browser history and your favorites and populates a flyout with relevant searches. In other words, it’s a quick interface to search for relevance among things you’ve already flagged as relevant by visiting or starring. And when I say it searches through your browser history, I don’t mean it looks for strings in the url. It does a full text search on pages you’ve visited. So even if you didn’t star that page about platypuses that you found so interesting, if you want to get back to it without combing through thousands of results from Google, you can just start typing “platypus” in the search box, and sites you’ve visited that contained the word will magically appear in the flyout.

Next, there’s photo integration. We have this thing called a topbar that’s a little slice of screen that pops open between the browser buttons and the content window. It’s a pretty good size for displaying thumbnails of photos, and it’s got a neat little slidey interaction that makes it easy to zoom through photos it contains. The photos currently can be set up to pull from accounts at flickr and photobucket, and you can use the topbar to view your own photos or those of other users. The topbar will also notify you by lighting up a browser button when your contacts post new photos, so there’s no more running out to click through your contacts’ pages to see what’s new — Flock just brings it all to you with no effort on your part. This functionality is of pretty limited use to me because I don’t follow photos very closely, and the ones I tend to care about usually get posted to kodakgallery, which Flock doesn’t yet support and which sends me email invitations to see photos anyway. What is useful to me to the extent that I use it is the fact that Flock comes with a photo uploader that makes it pretty easy for me to send photos to flickr. As a Linux user, I’ve always had to do it the painful way, using file upload widgets and navigating through files by name to upload. This is definitely not a good way to batch upload photos. Now I can just open the uploader, browse files or drag files to the window, rotate and crop, and upload. If I were inclined to use photo services very frequently, this would be a great tool for me, especially since most of the tools out there that simplify photo uploading don’t work in Linux. Rumor has it that you can also now drag photos into textareas in the browser and they’ll get uploaded and posted automatically, but I haven’t tested that functionality out yet and so can’t verify that it works.

The last two features that stand out to me are things that used to excite me but that now completely underwhelm me. The first is the blog editor. It’s built right into Flock and lets you edit and save posts locally before pushing to your blog service. One nice thing is that it’s got drag and drop goodness, and I’ll bet that by this time, if you drag photos into it, they get pushed to your photo hosting service. Generally, I’m about as happy editing blog posts in my blog’s user interface, and as far as blog editing usability goes, I prefer the extension produced by Performancing to our blog editor. This is really sort of a shame, as blog integration was one of the things we wanted to address best and first, and in my opinion, we’ve regressed a bit on that front. The other feature I used to like that I now find almost useless is what we’re calling web snippets. This used to be a little sidebar or floaty window that you could drag photos and text snippets to for later use in blog posts. We later added the ability to add notes to the web snippets, and I found this useful as sort of a blogging to-do list. But we crammed the tool down into a little area at the bottom of the screen and made it pretty much unusable because of the way it uses its real estate. In theory, it helps you assemble things to put into blog posts, but in reality, for me, at least, it sits down in the status bar unused. These two features were very appealing to me when we started working on them around a year ago, but they’re now the least useful things we’ve added to the browser.

Which is ok, because, as I think I’ve indicated, we’ve added other useful things. The news functionality alone makes me want to use Flock. I think we’ve also produced a beautiful browser. As much credit as Firefox is due for helping to bring choice back to the web browser industry, the browser’s default theme is heinous. Any time I run Firefox now to test something, I’m shocked at how clunky the buttons and toolbars look, and I find it a little oppressive. Using Flock just feels smoother to me because our interface looks smoother. This is of course purely a matter of taste, and it should be noted that both Flock and Firefox can be re-skinned to look entirely different and better (or worse). I think that for the time being, Flock provides a better experience out of the box, though.
So, that’s what I’ve been doing for the last 18 months. I do primarily web programming and system administration these days, though I’ve been involved at some point in pretty much every part of the business. Last week, we released our first public beta (we’ve been in developer alpha for months), and feedback so far is pretty good. We’re still recommending that people try Flock out but not be disappointed if we’re not quite ready for prime time or if the browser’s a little buggy. If any of the features I’ve described sound useful to you, I invite you to try the browser out. You can download it here. If you’re not game, no hard feelings; as I said at the beginning of this post, Flock isn’t for everybody.

Note: I’m cross-posting this from my Flock blog. We sometimes call our staff and users flockers. I post here for the obvious relevance to Lennie, and I offer this explanatory note because I refer to her in more distanced than usual terms like “my daughter.” Without further ado…

Huh? My daughter woke up at 4:30 this morning, and I was treated to very early morning TV. We often tune in to a station called Noggin that’s got some really great shows (take that, Barney). I like PBS and all, but it doesn’t hold half a candle to Noggin. I usually get my daughter up around 8:00 or 8:30, and we’ll tune in to Max and Ruby or Little Bear (I like Noggin, but both of these shows annoy me) while we eat some breakfast, take our Flintstones vitamins together, etc.  Noggin only runs from 6:00 a.m. to 6:00 p.m., so we had to entertain ourselves for the first hour-and-a-half this morning (during which time mostly I lay in a daze on the couch while my daughter piled toys on me). When the cartoons started rolling, I was treated to a new (to me) show called Tiny Planets (review). The basic premise is that two fluffy white aliens fly around on their fluffy white couch to various planets in their area. One is the planet of light and color; another is a music planet; another is a nature planet; I think there are six in all. As they catapult around (literally — their couch is slung from their home base by a huge catapult attached to a cord by which they’re ultimately reeled back in) to the various planets, they experience various adventures and misadventures that afford them ample opportunity to try to use critical thinking skills to get out of the jams they find themselves in. It’s a neat little show, a CGI cartoon that’s wacky and strange, but fun.

A couple of times, as they were sling-shotting through space, they passed an asteroid on which three creatures were running around (that is, they were running in place with the asteroid spinning under their feet). The first time the narrator pointed them out, I wasn’t sure what she called them. The next time, I thought it sounded vaguely like “flockers.” And then a third time, in reference to similar creatures on one of the planets, I was pretty sure that’s what they were called. And sure enough, when I looked it up, I learned that these critters are called flockers. They’re described in an epinions review of the show as “a bird-like alien whose intelligence can best be described as ‘dim.’” Luckily, our own flockers far outpace these guys.

So, there’s not much of real relevance to Flock here, but this was a fun little discovery. If you’ve got kids and are fed up with Barney and the other usual suspects, check out the Noggin web site and consider queuing Tiny Planets up in your TiVo if you’re not game for an early morning with your little ones.

Almost exactly a year ago, I wrote about my worries pertaining to my first transnational trip. I’ve come a long way since then. Some forms of mass transportation still frighten me. I worry about ever having to ride a train on my own, for example, because what if I miss my stop and have to do the whole circuit again and then am so embarrassed and feeling provincial enough that in my ducking and hiding away, I miss my stop again on the next circuit and eventually, I wind up riding for six or eight hours that I don’t have enough cash to pay for and I have to take a job as a ticket-puncher on the train to work off my debt and this exascerbates my feelings of stupidity and provinciality and I’m afraid to go back to my office when I finally work off my debt because after all I’ve missed most of a day of work doing something pretty dumb and so, well, that’s why I’m not taking the train into San Francisco any time soon.

But air travel I’ve pretty much got down pat by now.

I’m working on a community dashboard for work that’ll help me monitor different ways in which people are interacting with the community. One metric I thought it’d be neat to have was how many people have filed more than X bug reports during the last period Y (month, say, or week) and what the average for that period over time has been. This involves a pretty complicated query wherein you have to group by both user and period, which is kind of tricky. It’s also very expensive: With only about 1200 bugs in our database, this query was taking six seconds when I was testing it out. That’s clearly unacceptable. So I poked around a bit and discovered the GROUP_CONCAT function, which lets you group by one column and output the relevant grouped data in a delimited column. So say your query is something like this:

select DATE_FORMAT(creation_ts,’%Y-%m’) date, group_concat(reporter) cnt from bugs group by DATE_FORMAT(creation_ts,’%Y-%m’)

Here I’m getting all months in the database and a listing of all reporters of bugs for that month. The “cnt” column is a comma-delimited lists of user ids for those who reported bugs in the given month, duplicates included (though you can specify DISTINCT to eliminate duplicates). The query returns almost instantly and gives me something I can parse pretty easily in the programming language of my choice.

In my case, I split the “cnt” column on commas for each result and tally the ids for users who appear in the list X or more times. I then pass this data back to a function that does math to get averages and to find a count for the current month, and voila, community metrics.

I blog this here because it’s the sort of thing I might want to remember later, and I always put that stuff here rather than at my work blog, where my regular readers probably figure this should actually go.

As noted previously, I’ve been overwhelmed by the positive response Flock has gotten so far. The community forums have been hopping (I’m trying to put a little dent in responding to a backlog each day; if there’s something pressing in there that nobody’s responded to yet, drop me an email at daryl at flock.com and I’ll see if I can help), and we’ve had a good bit of activity on our Flockstars list. We’ve also had an overwhelming number of offers of help. Believe it or not, it can be very hard to accept help because all the offers have to be organized, but don’t worry — we intend to organize all the offers and accept a lot of the help.

For the time being, if you’d like to volunteer to help out in some way, you can go here to do so. We might not get back to you right away, but by submitting your info here, you guarantee that it gets stored in a searchable database rather than lost in somebody’s inbox or blog comments.

If you’re one of the few dozen people who commented on Geoffrey’s blog post, I’ve probably sent you an email asking you to do the unspeakable and resubmit your info using the new page. I hate to ask you to do this, but it saves me a lot of time that I can put to better use actually working on Flock, and I figure you guys’re all behind that and will forgive my asking you to do something sort of redundant.  If you’ve offered through some other means, I hope you’ll also do me the favor of filling out the new form with your info, just to make sure we’ve captured your offer. Thanks!

Technorati Tags: flock, work, tech

On Monday, I had my gall bladder removed. Yesterday, I worked a 15-hour day because we decided to launch Flock publicly. Today, I’ll be good for more or less nothing.

We’ve been releasing to larger and larger beta groups over the last week or so, and we decided yesterday afternoon to go ahead and release publicly. Of course, we had much work do to, as we had been focusing more on shipping a browser than on getting our public Web site in good shape, etc., and so our crew spent the evening whipping that stuff into good — not great, but good enough — order for public consumption. This meant crossing our fingers that we could withstand a slashdotting better than we did two weeks ago (because of some heavy-duty server infrastructure changes we made, we weathered it without issue). This meant working on our messaging a bit in order to help preempt some of the stupid sorts of comments that always seem to emerge. When we half-released for Web 2.0 a couple of weeks ago, for example, rumors abounded that we were taking Firefox, making it Windows-only, and slipping in adware. WTF? So we obviously wanted to have some blog posts and FAQs in place to explain some of what we’re doing. What we’ve got so far is available at www.flock.com.

The slashdot reviews have been much more even-handed this time. There have been the expected trolls (and let me just say that even though the trolls shouldn’t bother me, they do, because after all, I’ve missed bits of my daughter’s childhood to write the software they’re panning, and I’ve sat up with a hurting abdomen the day after surgery to do things to get a pre-release out; the whole crew has made big sacrifices to ship the software these trolls dismiss out of hand without regard for the human effort that drives its production; one would hope they’d at least download the software and try it out before dismissing it, but it seems that few do). But there have been some really good comments as well. Among my favorites:

Giving me quick access to something like a blog or Flickr isn’t “innovative”. A bookmark/favorite does the same thing with less overhead.

I thought the same thing until I actually tried the Flock Developer Preview that was just released. (I’m posting this from it now.)

I was all set to be unimpressed but I have to tell you, it’s pretty impressive if you have a blog how easy they have made posting Web content to it. There’s a “shelf” tool, for starters, that you use by just highlighting any text on a page and dragging-and-dropping it into the Shelf. Then, when you want to post about that text, you just click the “Blog this” button on the toolbar; this opens a new post (Flock autodetects the settings for your blog, so there’s no configuration if you use most popular packages) in a WYSIWYG editor. Drag the text from the shelf into the editor and it pops the text in, encloses it in BLOCKQUOTE tags, and adds the cite=”" attribute with the URL from the original page.

Revolutionary? Maybe not. But it’s so damn slick! Currently when I blog something I copy it from Firefox into an HTML editor (Movable Type’s built in editor sucks), mark it up there, log into the admin screen for my blog, then paste the marked-up text into a new post. Oh, and then I have to go back and find the original URL, copy it, and paste it in the appropriate pages. That’s a lot of back and forth that Flock eliminates.

Some people use a tool like MarsEdit [ranchero.com] or wBloggar [wbloggar.com] to combine the “markup” and “posting” steps together in one place. But Flock puts all the features of those products right in my browser — no switching between programs, no copy/paste gymnastics. There’s a market for those products, so it’s not a big leap to imagine a market for Flock, either (albeit a small one).

It’ll be interesting to see how well Flock holds up to ongoing use over time. But my first impressions are better than I expected them to be. You might want to try it too before you pass judgement…

Slashdot | Firefox-based Social Browser Flock Launches

and

Flock had me skeptical from the screenshots (ugly and useless), but having actually used it, it’s pretty gosh darn neat. The Shelf is an incredible killer feature. I’ve tried out a few similar extensions for Firefox, but none did it as smoothly and intuitively as Flock has. All it needs is a few hardcore snippet-management-tools, and it’ll be my new favorite research program.

Likewise, the blog editor falls under the “pretty neat” status. The formatting gets eaten by Wordpress.com’s post-parser (to filter out nasty javascript and other malicious evil), but that isn’t a major downer, as it does tend to exhibit some weirdness like underline remaining after deleting a link. The WYSIWYG-editor part of it definately needs some work to be up to par with the rest of the browser.

Overall, I’ve been seriously impressed. For being a the first public release of a browser, it’s feature-filled and non-crashy. This must be attributed to it being based on Firefox. All it needs is a few months of polish and I can unconditionally accept it as my new primary browser. As is, I’m giving the idea serious thought.

P.S.: I didn’t use the del.icio.us integration, as I didn’t really use the service much before. But now that it’s seamlessly integrated into the browser, I’ll try it out again.

Slashdot | Firefox-based Social Browser Flock Launches

Of course, these appeal to me in part because I’m the perpetrator of the shelf (with its various warts and pimples), which gets pretty good reviews so far. But they also give me a little hope for the slashdot crowd. There actually are those who expect one thing, evaluate the software, and come away with a more balanced view rather than falling back on the old standby comments about how silly social networking is or how we must be making Firefox into a Windows-only spyware engine.

It’s been a good morning. I’ve been reading and responding to mail from our flockstars mailing list, where people are already submitting extensions that will work in Flock, and glancing over the hundreds of feedback emails that have come in over the last 12 hours. Response overall has been very good. Yes, we have a long way to go, and there are many crucial bugs we have yet to fix, but the people we’re aiming at seem to think we’re on to something, and that’s nice validation.

I plan to work until noon today and then to take most of the rest of the day off to try to spend with my daughter, whom I don’t think I even hugged or kissed once yesterday. Then there’ll probably be a regrouping/evaluation conference call this evening where the Flock crew will talk about the last day and the coming very busy days, deciding just how we’re going to keep up our momentum and try to deliver a better and better and better product. Stay tuned!

Technorati Tags: flock, work, tech

People I know seem not really to know what I do for a living. They know it has to do with computers, and most of them know I’m working on
software, and many know that I’m doing something with browser software. In a nutshell, here’s what my company does:

Flock hopes to turn the browser into a dashboard for collaborating, blogging, sharing photos, reveling in a raft of other group activities that have recently caught fire online (see BW, 9/26/05, “It’s a Whole New Web”).

Flock, the New Browser on the Block

So far, based on my dogfeeding our software last night and this morning, we’re well on the way to success. I’m blogging manically lately, I guess. What we’re doing is exciting, and it’s good to have something techy to blog about, plus it’s cool to use the tool I’ve helped create to blog about said tool. My apologies (to both of my readers) for all the redundancy lately.

Technorati Tags: flock, work