Posts tagged "Hacker-School"

Blogging catch-up

Greg Hendershott — Sat, 06 Dec 2014 01:00:00 UT

Blogging catch-up

2014-12-05 :: Hacker School

It’s been a few weeks since I’ve blogged. Bad me. This is a catch-up post.

Hacker School has a tool called Blaggregator created by Sasha Laundy. We can submit feeds for our blogs. Blaggregator provides an aggregate page, and puts new-post messages on Zulip, the chat tool.

Awhile ago I noticed a bug with time zones in the RSS feed generated by Frog. I was concerned that fixing the feed would change date-times, and that might cause Blaggregator to flood Zulip with “new post!” notifications. Which would be embarrassing. So I unsubscribed my feed, fixed the problem, and subscribed again. All went well, but Sasha pinged me on Zulip today to follow up and make sure.

That prompted me to check out Sasha’s profile page and list of projects. That led me to On Rakefiles and Rabbit Holes. This is such a classic Hacker School type of post — where someone takes the time to follow the trail wherever it leads, and learn cool new things.

Sasha’s mention of stty -echo reminded me of a mild annoyance in Emacs’ shell-mode. Sometimes — but not always — it starts echoing my input, as if I had typed stty echo. Sasha’s post this reminded me I could type stty -echo to turn it off. But how and why does it turned on in the first place? I searched and found a Stack Overflow question, Python in Emacs shell-mode turns on stty echo and breaks C-d.

Aha. As it says:

OS X ships with the BSD-licensed editline rather than the GPL-licensed readline, so this would explain why the behaviour is different on OS X from other Unixes. The same thing happens with other interactive interpreters. I find that the Lua, Ruby and Sqlite3 command-line interpreters also turn on terminal echo when run inside Emacs. So it seems to be some kind of “feature” of the editline library.

Yes. A fresh M-x shell is fine, until I run (say) python or racket.

Anyway, Sasha’s Zulip message managed to nudge me into writing a blog post.

So what have I been up to?

My first six weeks here, I mostly spent multiple weeks at a time on things like getting hands-on with Clojure and then Haskell, or pairing a full week on a web service.

Whereas so far in the second half, I’ve been doing a variety of smaller projects — which I haven’t blogged about one by one. Here’s a quick overview:

Help some people who were pairing to create a lean web framework in Python. We got to see first-hand how cookies are a way to provide state for a multi-page form application.
Work through the Rust Guide hands-on.
Pair with someone working on a Python web app that shows you if it’s Shabbat in the location of someone you want to call. They wanted to refactor the code to be less repetitive.
Participate in Zach Allaun’s hands-on workshop about configuring and developing using ClojureScript.
Fix a bug in racket-mode on Windows.
Pair with someone who is vastly more experienced in Haskell than me, to learn how to use Template Haskell for my Haskell port of wffi.
Fix some issues with a couple of my Racket packages, aws and http, that are now being used by Racket’s DrDr continuous integration tool.
Pair with someone who is getting started with Racket and wanted to use racket-mode on Emacs.
Start to learn about microKanren from looking at sokuza-kanren.scm and a couple papers including μKanren: A Minimal Functional Core for Relational Programming.
Start to work through the Matasano crypto challenges.¹

In most of these cases, I didn’t feel like I had an entire interesting blog post to write. I did take notes and had some fantastic experiences both learning and teaching.

All the friends and family who think I’m learning cracking here at Hacker School? This is the closest I’ve come so far. ↩

Hacker school week 6

Greg Hendershott — Fri, 14 Nov 2014 18:42:14 UT

Hacker school week 6

2014-11-14 :: Hacker School

I’m at the end of week 6 at Hacker School, which marks the halfway point. There are overlapping 12-week batches. As a result, the previous batch never-graduated yesterday.

My early weeks here included some pairing, and it was fun, but I spent more time learning and coding solo. My previous week was nearly the opposite. I spent most of my time pairing with Sumana Harihareswara.

Her project idea was a web service that would help people find vulnerabilities in C code, and return a report card itemizing the issues and giving a score¹.

Writing a static code analyzer from scratch seemed like it would be too ambitious for the time available. So for version 0.1 we decided to have our web service wrap existing tools like Clang Static Analyzer and Cppcheck. Someone can use the analyzers without needing to install them.

We installed and got familiar with clang’s scan-build. I wrote some C code with deliberate problems, and we saw what was reported.

For our web server, we decided to make something using Python. We started to look at our choices for frameworks, like Django, Flask, and several dozen others. But then we realized, hey — our web service is really simple. It has one resource. A request will be:

POST /api/analyze HTTP/1.1
Content-Length: 42

/* my awful C code */
main () {
}

Our response body will be scan-build output parsed into some sort of JSON.

Do we need any framework at all?

We decided no. We derived from BaseHTTPServer, overriding do_POST. This became a great opportunity to look at how HTTP requests and responses work.

Of course, if our web service were to grow, we would find ourselves copying and pasting a bunch of code. We would DRY it into some helper functions. One day we’d say, “Let’s share this with the world!”. And voila, the world would have another framework.

Or better yet, we’d say, “OK, now let’s use some existing framework. Now we know what problems it’s solving for us. Now we have a reasonable mental model for how it probably works; it’s not some magical black box.”

When it came time to parse the scan-build output into JSON, I found it quicker to write that in Racket than in Python. Which was fine, because that was something we needed done, as opposed to something we wanted to learn about.

At this point we had a reasonably well-working application that ran… on our laptops. We wanted to deploy it on a Digital Ocean droplet. Putting one web service on one droplet isn’t difficult. But we opted to pretend that our web service would become wildly popular and we’d need to scale it across multiple machines. A couple people had asked if we were going to try Docker. We did.

It took us awhile to wrap our head around the concepts. Eventually we understood that an image is a read-only thing created from a Dockerfile. It is built from a base image, e.g. Ubuntu, plus additional packages that you install, and other programs that you run and settings that you configure. Thereafter you launch containers for an image. These have a read/write layer and are a kind of lightweight VM in which your application runs. You can docker run -d <image> to detach these and they run in the background; you docker logs <container> to see their output. You can docker stop <container> to stop one. After stopped it still exists. You can see all these with docker ps -a, or just the running ones with docker ps. You can delete a stopped container with docker rm <container>.

Therefore we wanted to create an image based on Ubuntu, install some packages like git and Racket 6.1, and finally do a git clone of our app code.

A couple issues that slowed us down briefly:

apt-get wouldn’t work until we changed /etc/default/docker/ to un-comment the line saying to use Google’s DNS servers.
How to set an environment variable in our Dockerfile, that would survive into the container’s environment. Doing RUN export PATH=$PATH:/new/path did not work. Turned out we needed ENV, as in ENV PATH $PATH:/new/path.

In the end we got things working.

The final day we made a simple web site to go along with our web service. So as a human you can visit a page, type C code in a form or do a file upload, and get the report card on a web page rather than as raw JSON. Although we didn’t learn a whole lot by doing this part, it was a satisfying way to wrap up the project.

I enjoyed the project because I got a chance to:

Try more Python coding in a week than I’ve done, ever.
Understand Docker hands-on.
Exercise developer teamwork skills.
Validate my understanding of HTTP mechanics by explaining them effectively.

For my second half of Hacker School, I’m looking forward to doing more pairing. After all, I can study things solo anytime. Now is when I have so many great opporunities to pair.

Silicon Valley has its “Weisman Score”. Well this is a “Hendershott-Harihareswara Score”. Really rolls off the tongue, doesn’t it? ↩

Hands-on with Haskell week 1

Greg Hendershott — Mon, 03 Nov 2014 15:51:21 UT

Hands-on with Haskell week 1

2014-11-03 :: Haskell, Hacker School

Last week I decided to pivot from Clojure hands-on to Haskell hands-on.

Rewiring brain from Lisp to Haskell

Prior to this, I had spent some time working through examples in Learn You a Haskell and armchair browsing Real World Haskell. Also I’d spent some time using Typed Racket and was already sold on the benefits of having a “real” type system, especially for certain sorts of programs. So this wasn’t starting from scratch. However I soon discovered there were some things about Haskell that I’d forgotten and needed to re-learn. Also I suffered from a Lisp brain configuration.

Sections

One example was sections. On Zulip I saw a code fragment like this:

1	primes = 2 : [i \| i <- [3..], and [rem i p > 0 \| p <- takeWhile ((<=i).(^2)) primes]]

With the comment that the (<= i) was equivalent to (\j -> (j <= i)).

Wait. Why not (\j -> (i <= j))?

The answer is that a section behaves differently when used with an infix operator. The argument is applied on the missing side. In other words (/ 10) and (10 /) are different.

So (< 1) 2 is False but (1 <) 2 is True. And (compare 1) 2 is True.

In Haskell it’s not the case that (< 1) is 1 partially applied to the < function, waiting to take a “right” argument.¹

Commas

In working through Learn You a Haskell I wanted to make an example input to try with nub. I wanted a long list containing duplicates, that I didn’t have to type in by hand. To do so I wanted to do a Racket flatten of a Haskell replicate of a short list. It seemed like foldr1 with ++ would be the way to do the flatten. So I tried something like:

*Main> foldr1 ++ (replicate 5 [1 2 3 4 5])

<interactive>:107:1:
    Couldn't match expected type `[a0]'
                with actual type `(a1 -> a1 -> a1) -> [a1] -> a1'
    In the first argument of `(++)', namely `foldr1'
    In the expression: foldr1 ++ (replicate 5 [1 2 3 4 5])
    In an equation for `it': it = foldr1 ++ (replicate 5 [1 2 3 4 5])

Oops. Let’s try something simpler, first.

*Main> foldr1 ++ [[1][2]]

<interactive>:245:1:
    Couldn't match expected type `[a0]'
                with actual type `(a1 -> a1 -> a1) -> [a1] -> a1'
    In the first argument of `(++)', namely `foldr1'
    In the expression: foldr1 ++ [[1] [2]]
    In an equation for `it': it = foldr1 ++ [[1] [2]]

Huh. Oh right. The ++ is infix. Needs to go in parens. Try again:

*Main> foldr1 (++) [[1][2]]

<interactive>:246:14:
    The function `[1]' is applied to one argument,
    but its type `[t0]' has none
    In the expression: [1] [2]
    In the second argument of `foldr1', namely `[[1] [2]]'
    In the expression: foldr1 (++) [[1] [2]]

Bzzzt. Tried to decipher the error message. Finally remembered, dang, I need to separate list items with commas:

1 2	*Main> foldr1 (++) [[1],[2]] [1,2]

OK. And building back up again to what I’d wanted originally:

*Main> replicate 5 [1,2,3,4,5]
[[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5]]
*Main> foldr1 (++) (replicate 5 [1,2,3,4,5])
[1,2,3,4,5,1,2,3,4,5,1,2,3,4,5,1,2,3,4,5,1,2,3,4,5]
*Main> nub $ foldr1 (++) (replicate 5 [1,2,3,4,5])
[1,2,3,4,5]

In Lisps I’ve grown accustomed to — and really love — being able to separate list items with whitespace. Needing to use commas feels a bit weird. Also it’s a bit weird that you only use them for things like lists and tuples, but not in function applications. Oh well.

Other tiny things

I keep making the mistake of typing data MyType = _ instead of data MyType = MyType _.
Racket structs define accessor functions prefixed with the name of the struct. So (struct person (name)) defines a person-name accessor function. In Haskell, data Person = Person {name :: String} defines an accessor name — no prefix. Problem being, what if you have a couple types that have fields called name? I guess you have to do prefix the field names explicitly yourself, e.g. data Person = Person {personName :: String}?

None of these observations are intended as a critique of Haskell. It’s an observation how the tiny things can trip you up when learning a new language. Even after a couple days, I’ve developed a sort of subconscious checklist of rookie mistakes to consider before trying to understand the Haskell error message.

Nice things

I like how creating a simple Haskell “project” (just for local use) is as simple as creating a .hs file. I also like this in Racket.
I like having an actual, mature module system with clear best practices (as in Racket).

My first hands-on with Cabal

I used Cabal for the first time, to install Pandoc. This took… awhile. Finally I got a weird version error:

Configuring pandoc-1.13.1...
setup: At least the following dependencies are missing:
http-client >=0.3.2 && <0.4 && ==0.4.2.2
Updating documentation index /Users/greg/Library/Haskell/doc/index.html
cabal: Error: some packages failed to install:
pandoc-1.13.1 failed during the configure step. The exception was:
ExitFailure 1

After puzzling this over for awhile, and reading up on Cabal versioning, I didn’t have any great ideas. Finally I decided to try running cabal install pandoc a second time. And this time, it worked. Shrug. Onward.

Porting wffi to Haskell

So after some “warm-up” I decided to dive in and try to port wffi to Haskell. This turned out to be a good-sized project when I was hands-on with Clojure — not too small, not too big. Implementing it requires parsing markdown, parsing HTTP request templates, making HTTP requests, using higher-order functions, and so on.

I forged ahead and got the wffi port of Haskell working to the point that I could successfully make a request to http://horseebooksipsum.com/. I had a little confusion at first about how HTTP.Network.simpleHTTP returns IO (Either String String). Basically, I didn’t anticipate where the IO vs. pure “boundary” would need to be in my program. But once I realized that, it made sense and was easy to sort out.

I pushed my commits to a GitHub public repo, even though I’m sure the code smells pretty badly. My normal instinct is to spend more time with code — beyond the “it’s amazing that the dancing bear dances at all” stage — before pushing to public. But I’m at Hacker School, this isn’t professional coding, and I shouldn’t be so cautious. In fact, by pushing early, I could get some feedback.

Surprisingly easy

I was surprised by how quickly the port to Haskell went, compared to the port to Clojure. I would have guessed the opposite, because Clojure seems like it would be “closer to” Racket. I think there are two reasons why it was faster.

One is that I hadn’t worked on wffi for a year, so part of the Clojure port was actually reminding myself how things worked. In other words I’ve been “practicing” porting wffi, and the Haskell port got the benefit of that.

But also, I simply found Haskell easier to use than I expected. Sure, I frequently alternated between 10 minutes of smooth sailing and 5 minutes of trying to decipher a type error message. However, this felt different from being stalled on a Clojure build/environment/package issue. The Haskell type error messages felt like they turned out to be useful information about my reasoning about my program. Resolving them was sometimes a matter of fixing a simple mistake. Sometimes it entailed fixing a problem with my thinking and my design.²

My early days with Haskell have been more satisfying than those with Clojure. Whether that continues remains to be seen. So far so good.

It’s also not the case in Racket or Clojure. You’d have to say (curry < 1) or (partial < 1). But if those languages did automatic partial application, it would look like (< 1). At least that’s what my Lisp brain misunderstood at first. ↩
This is an experience I also had with Typed Racket. ↩

Applicable symbols

Greg Hendershott — Mon, 27 Oct 2014 16:29:15 UT

Applicable symbols

2014-10-27 :: Clojure, Hacker School

Here are my notes about being puzzled about some Clojure code and diving into the implementation to figure it out. Although I figured it out the hard way, the exploration turned out to be interesting for me.

As I was reading Geoff Shannon’s blog post, I was thinking, “Oh, maybe this will turn out to be a mixup between compile time and run time”. (At least, in my experience writing macros, it can be challenging to keep that straight.)

Instead, what confused me was line 103 in the last example (here I’ve changed the symbol from '+ to 'x):

1	(apply 'x [1 2]) ;=> 2

Huh? I don’t understand. Why does this return 2, instead of raising an error?

First, let’s factor out apply and confirm the same result for a direct application:

1	('x 1 2) ;=> 2

Same result. Same mystery.

Let’s explore some examples:

('x)       ;=> ArityException Wrong number of args (0) passed to: Symbol  clojure.lang.AFn.throwArity (AFn.java:429)
('x 1)     ;=> nil  ???
('x 1 2)   ;=> 2    ???
('x 1 2 3) ;=> ArityException Wrong number of args (3) passed to: Symbol  clojure.lang.AFn.throwArity (AFn.java:429)

(type 'x) ;=> clojure.lang.Symbol

There’s another mystery — the arity 1 case of ('x 1).

To summarize so far: It appears that it is not an error to use a symbol in the function position of an application if the arity is 1 or 2.

I’m aware that Clojure collections are applicable. And I’m aware that Clojure tends to nil pun. So the arity 1 case might be less surprising… except that neither 'x nor 1 is a collection, so this seems like it should raise an exception, not evaluate to nil. And I don’t understand the arity 2 case.¹

What to do? I could ask for help. But let’s take this as an opportunity to go spelunking in some Clojure implementation code I’ve never seen before.

From looking at the source for clojure.lang.AFn, it seems that all invoke members call throwArity. It’s been some years since I did C++, and I’m not up-to-speed on Java. But if I understand correctly, AFn is a sort of default class that always errors. Applicable things (functions, keywords, collections) probably derive from AFn and override invoke members to do something other than error. Somehow such a class is created for the symbol 'x, and I’m guessing that it overrides the arity 1 and 2 variants of invoke.

That seems plausible. So, how to find this in the Clojure source? Instead of searching the code on GitHub, let’s git clone the source locally and use Emacs rgrep to search for invoke. I’m sure there are Java source navigation tools, but probably not worth choosing/installing/learning for something this straightforward. Yes rgrep returns 1200+ matches, but flipping through them quickly there are some obvious patterns — huge swaths that seem OK to ignore, and a few names that pop out.

For example in APersistentMap.java I see arity 1 and 2 definitions of invoke:

public Object invoke(Object arg1) {
	return valAt(arg1);
}

public Object invoke(Object arg1, Object notFound) {
	return valAt(arg1, notFound);
}

That sure seems how ({:key val} :key) and ({:key val} :key default) would work. valAt is a core member function that does the lookup of a key and value in the map.

OK. Next let’s try to find a class that might correspond to a symbol like 'x and examine its invoke member(s).

And bingo, here’s a file called Symbol.java, also defining arity 1 and 2 variants of invoke:

public Object invoke(Object obj) {
	return RT.get(obj, this);
}

public Object invoke(Object obj, Object notFound) {
	return RT.get(obj, this, notFound);
}

So this seems to explain why arities 1 and 2 are special. Next question: What is RT.get? In C++ this would mean a static get member of an RT class. That seems to be the case here, looking in RT.java. get comes in both arity 1 and 2 flavors (not counting the first argument; this is a static member so the implicit this argument is explicit). First here’s arity 1:

static public Object get(Object coll, Object key){
	if(coll instanceof ILookup)
		return ((ILookup) coll).valAt(key);
	return getFrom(coll, key);
}

Which of the two paths here would 'x take? I’m going to make a guess that a symbol probably isn’t an instance of ILookup, and what’s happening is the call to getFrom. Which is defined in RT.java as:

static Object getFrom(Object coll, Object key){
	if(coll == null)
		return null;
	else if(coll instanceof Map) {
		Map m = (Map) coll;
		return m.get(key);
	}
	else if(coll instanceof IPersistentSet) {
		IPersistentSet set = (IPersistentSet) coll;
		return set.get(key);
	}
	else if(key instanceof Number && (coll instanceof String || coll.getClass().isArray())) {
		int n = ((Number) key).intValue();
		if(n >= 0 && n < count(coll))
			return nth(coll, n);
		return null;
	}

	return null;
}

My guess is that all of the conditional branches fail for a symbol like 'x, and getFrom returns Java null. Which becomes Clojure nil. First mystery solved.

How about arity 2? This is the case where a “not-found” value is supplied:

static public Object get(Object coll, Object key, Object notFound){
	if(coll instanceof ILookup)
		return ((ILookup) coll).valAt(key, notFound);
	return getFrom(coll, key, notFound);
}

static Object getFrom(Object coll, Object key, Object notFound){
	if(coll == null)
		return notFound;
	else if(coll instanceof Map) {
		Map m = (Map) coll;
		if(m.containsKey(key))
			return m.get(key);
		return notFound;
	}
	else if(coll instanceof IPersistentSet) {
		IPersistentSet set = (IPersistentSet) coll;
		if(set.contains(key))
			return set.get(key);
		return notFound;
	}
	else if(key instanceof Number && (coll instanceof String || coll.getClass().isArray())) {
		int n = ((Number) key).intValue();
		return n >= 0 && n < count(coll) ? nth(coll, n) : notFound;
	}
	return notFound;

}

Aha. ('x 1 2) is being treated as, “Look up 1 in 'x and if not found return 2”.

Editorializing

It makes sense to support a symbol in the function position of an application to enable doing this:

1
2
3

('key {'key 42})    ;=> 42
('key {'key 42} 42) ;=> 42
('key {}        42) ;=> 42

Of course. That’s awesome.

But when the second element is not a collection? It should error. Looking up a key in an integer doesn’t make any sense. It is the result of a mistake. Clojure should tell you it’s an error. Returning something here is as bad as silent coercions in Javascript.

Postscript

For what it’s worth, here’s how this works in rackjure:

#lang rackjure

('key {'key 42})           ;=> 42
('key {'key 42} #:else 42) ;=> 42
('key {}        #:else 42) ;=> 42

('key 1)
; applicable-dict: No dict? supplied
; in: ('key 1)
; Context:
;  /tmp/foo.rkt:1:1 [running body]

One difference is that arity 3 means assoc. If you want to supply an explicit default, you use an #:else keyword argument.

The main difference is that a nonsense application results in an error.

Spoiler: By the end of this, I remember there’s an optional not-found argument. ↩

Hacker School day 15

Greg Hendershott — Sat, 25 Oct 2014 15:03:17 UT

Hacker School day 15

2014-10-25 :: Hacker School

I spent most of late Thursday and Friday working on open source projects that pre-date Hacker School.

Racket wanted to accept a pull request from me to fix a bug with small bit-vectors. But because I had stumbled around before finding the correct solution, and discussion on the PR, I’d accumulated about 8 commits on the PR branch that needed to be squashed. Also, the PR had been open for about 8 weeks, so I needed to fetch Racket HEAD and build in order to test properly. Finally I needed to wait for Travis CI to finish (the full builds of Racket there take ~30 minutes). In all, a fair amount of busywork, but that’s an important part of the process.

I fixed a bug in my markdown parser. After boiling it down to the minimal example, the problem was obvious; the parser was sometimes returning an x-expression at a stage where a raw string? was expected — in the case of an  inside a list item. The fix was simply deleting the check for the HTML comment. Which made me suspicious. Had there been some intent behind the check, even though it now seemed crazy? Or had the check just been a brain fart? I decided it was the latter, after thinking it through, and also seeing that all the unit tests (of which I have many) still passed. So I added the fix, plus a couple regression tests, committed, pushed, and commented/closed the issue on GitHub.

I fixed a bug in my racket-mode for Emacs. Turns out it wasn’t treating Unicode characters as symbol constituents, causing a problem with some paredit operations. Fortunately I had inherited the problem from scheme-mode, and knew that the problem did not occur in lisp-mode. As a result, it was straightforward to track down. The fix was simply setting multibyte-syntax-as-symbol to t.

I reviewed a pull request for #lang rackjure regarding how a Racket #lang could install a readtable for the REPL, without doinking the readtable for other modules.

Of the three bugs, two turned out to be one-line fixes — even though they took some time to figure out and to do “paperwork” like bug trackers and regression tests. On the one hand, such bugs feel like a low signal:noise ratio. On the other hand, they’re satisfying because they make you feel like the overall design/assumptions were good.

Also I did some work implementing Racket variants of some Clojure map-related functions like get, get-in, assoc-in, update-in, partition, take,¹ and juxt. More trivially, alias things like every?, some, assoc, and dissoc to their Racket equivalents. Although I haven’t yet pushed these to #lang rackjure I may do so over the weekend.

That was day 15 of about 60. On the one hand, it feels like a very long 3 weeks (in a good way). On the other hand, I can’t believe I’m already 25% through.

Time perceptions, how do they work?²

Unlike Racket’s, this doesn’t raise an exception when there are fewer than N elements left, it just returns them. Which IMHO is more useful; in the past I’d written a take<=. ↩
Also: Magnets. ↩

Why macros?

Greg Hendershott — Tue, 21 Oct 2014 17:56:54 UT

Why macros?

2014-10-21 :: Racket, Clojure, Hacker School

Yesterday a couple people asked me, “How and why do you use macros in a Lisp like Racket or Clojure?”.

I gave answers like:

The compiler can do a search-and-replace on your code.
You can make DSLs.
They’re an “API for the compiler”.

Although all true, I wasn’t sure I was getting the full idea across.

Worse, one time Peter Seibel was within earshot. Although I don’t know if he heard my explanation, I imagined him biting his tongue and politely remembering the “well, actually” rule. :)

Later I remembered Matthias Felleisen boiling down macros into three main categories:

Binding forms. You can make your own syntax for binding values to identifiers, including function definition forms. You may hear people say, in a Lisp you don’t have to wait for the language designers to add a feature (like lambda for Java?). Using macros you can add it yourself. Binding forms is one example.
Changing order of evaluation. Something like or or if can’t really be a function, because you want it to “short-circuit” — if the first test evaluates to true, don’t evaluate the other test at all.
Abstractions like domain specific langagues (DSLs). You want to provide a special language, which is simpler and/or more task-specific than the full/raw Lisp you’re using. This DSL might be for users of your software, and/or it might be something that you use to help implement parts of your own program.

Every macro is doing one of those three things. Only macros can really do the first two, at all¹. Macros let you do the last one more elegantly.

I think the preceding is a better answer. However, maybe it’s still not the best way to get people from zero to sixty on, “Why macros?”.²

Maybe the ideal is a “teachable moment” — facing a problem that macrology would solve.³ That’s also good because you really really really don’t want to use a macro when a normal function would suffice. So the goal isn’t to get people so enthusiastic about macros that they go forth in search of nails to which to apply that new hammer. Macros often aren’t the right approach. But once in a while, they are the bestest approach ever.

A language like Haskell can choose lazy evaluation, and implement if as a function. I’m saying that only a macro can futz with whatever the default evaluation order is, be it eager or lazy. ↩
Although I wrote a guide called Fear of Macros, it’s (a) specific to Racket macros and (b) much more about the “how” than the “why”. ↩
Certainly that’s my own optimal learning situation, as opposed to getting answers or solutions before I have the questions or problems. ↩

Hands-on with Clojure day 5

Greg Hendershott — Tue, 21 Oct 2014 17:54:46 UT

Hands-on with Clojure day 5

2014-10-21 :: Clojure, Hacker School

So I’ve fallen behind on the blogging, for a few reasons. Time to catch up.

I’m calling this “day 5” as a useful fiction. It’s a distillation of what is closer to days 5–7, or something like that.

As I mentioned before, this series of blog posts is going more directly from brain to web. Reflection and editing? Not so much.

Clojure port of wffi

I finished what I think is a reasonable initial port of wffi from Racket to Clojure. Pushed at clojure-wffi.

The simplest possible example is, given a horseebooks.md file like this:

# horseebooksipsum.com

Endpoint: http://horseebooksipsum.com

# Get

## Request
````
GET /api/v1/{paragraphs}
````

You can write:

1 2	(defwrappers "horseebooks.md") (pprint (get {:paragraphs 2}))

Which prints:

{:orig-content-encoding "gzip",
 :trace-redirects ["http://horseebooksipsum.com/api/v1/2"],
 :request-time 190,
 :status 200,
 :headers
 {"Content-Type" "text/plain",
  "Transfer-Encoding" "chunked",
  "Connection" "close",
  "Vary" "Accept-Encoding",
  "Cache-Control" "no-cache",
  "Server" "Apache/2.2.22 (Debian)",
  "Date" "Tue, 21 Oct 2014 18:07:16 GMT"},
 :body
 "Principle to work to make more money while having more fun. Unlucky people.
 And practical explanations. Process from preparation, through to delivery.
 And practical explanations. Process from preparation, through to delivery.
 And practical explanations. And practical explanations. Process from
 preparation, through to delivery. And practical explanations. And practical
 explanations. And practical explanations.\n\nDon't stubbornly. This is a
 very special technique that I have never seen. Don't stubbornly. This is
 a very special technique that I have never seen. And practical explanations.
 Don't stubbornly. Principle to work to make more money while having more fun.
 Unlucky people. Process from preparation, through to delivery. Don't
 stubbornly. Process from preparation, through to delivery. And practical
 explanations. This is a very special technique that I have never seen.
 And practical explanations. And practical explanations.\n\n"}

Of course this simple example doesn’t show much value-add. But real-world web services often have numerous parameters allocated among URL path segments, query parameters, and headers. With wffi, useful keyword wrapper functions are automatically generated from a markdown file that both documents and specifies the web service.

If I weren’t at Hacker School, I would spend much more time polishing and refining this. However this project is really just a means to the end of learning Clojure. So I’m going to force myself to task-switch to something else, next. I’ll return to this project if/when it seems like the best vehicle to learn more.

`split-with` and lazy seqs

Previously I posted that split-with seems to have an inefficient implementation.

Needing something like Racket’s splitf-at, I wrote a quick and dirty version in Clojure:

(defn split
  "FIXME: This is the conceptual, inefficient implementation. Should
  reimplement like Racket's splitf-at."
  [pred coll]
  [(take-while pred coll)
   (drop-while pred coll)])

This isn’t great because it traverses the first portion of the collection twice.

Someone pointed out that Clojure already provides this. It’s called split-with. Nice. But when I M-., I see that its definition is my conceptual one, not the efficient one.

Racket defines splitf-at like so:

(define (splitf-at list pred)
  (unless (procedure? pred)
    (raise-argument-error 'splitf-at "procedure?" 1 list pred))
  (let loop ([list list] [pfx '()])
    (if (and (pair? list) (pred (car list)))
      (loop (cdr list) (cons (car list) pfx))
      (values (reverse pfx) list))))

I “ported” this to Clojure like so:

(defn efficient-split-with
  [pred coll]
  (loop [ps [], coll coll]
    (if (and (seq coll) (pred (first coll)))
      (recur (conj ps (first coll)) (rest coll))
      [ps coll])))

One neat thing is the use of conj with a vector means we don’t have to do the reverse like we do in Racket, which should be even more efficient.

So why does Clojure implement split-with the way it does? David Nolen pointed out that I was forgetting about laziness. Aha.

In connection with this I learned about “chunked sequences” in Clojure from a Fogus blog post. Chunked sequences were added as an optimization in v1.1. The force granularity was increased from 1 item to 32.

Someone else pointed out that, had transducers been a thing, maybe lazy seqs wouldn’t be needed. (At least not as a default policy. You could have something like Racket’s racket/stream, with laziness and memoization.)

I already understood, in theory, that side effects expose the difference between eager and lazy evaluation. I learned, hands on, that this includes side effects like ad hoc debugging printlns.¹ For example if you have:

(let [coll (map (fn [x]
                  ;; 0: some bug that will throw an exception
                  )
                coll)
      _ (println "Everything AOK thus far -- or maybe not!")]
  ;; 1: use coll in way that forces the lazy seq
  )

The error won’t occur at 0, it will only occur at 1. The progress println will be misleading. At least it misled me, for awhile, about the actual location of a bug.

With Clojure and Haskell, I’ll need to keep in mind when and where laziness is used as the default policy. If I understand correctly, in Clojure that means lazy sequences, and in Haskell lazy evaluation generally.

Load vs. modules

After about a week hands-on with Clojure one of the things I miss the most from Racket is modules. Not even Racket’s submodules. Just plain modules.

Clojure namespaces handle name collisions. But modules go further:

Forward references are OK.
Redefinitions are flagged as errors.
Deleted definitions actually disappear from the environment on re-eval of the source file.

In other words, Clojure seems to be like Racket’s #lang racket/load, which isn’t recommended for general use.

An example scenario: I rename a function from foo to bar. I overlook updating a call site. Clojure doesn’t warn me with an error. Worse, I change bar’s behavior. But old foo still exists — and is being used at the overlooked call site. Hilarity and gnashing of teeth ensues.

This isn’t a hypothetical example. It’s happened to me a couple times in not that many days of hands-on with Clojure. On the one hand, this seems like an insane programming workflow. On the other hand, I have already learned to “measure twice, cut once” when renaming — and to bite the bullet and invest 10 seconds in a cider-restart. So if life had to be this way, I could cope.

But why does it have to be this way? I actually started to draft a module macro for Clojure. As I put in its README:

DISCLAIMER: This is a first draft by someone who…

has been hands-on with Clojure for just a week

doesn’t necessarily appreciate how Clojure namespaces work

doesn’t know the complete story behind Racket modules

I can imagine Inigo Montoya telling me, “I do not think module means what you seem to think it means”. Yeah. That’s probably me, with this code. At the moment it’s an exercise in starting to think about what might be involved.

Conclusions and next steps

There is a lot about Clojure that I really, really like and enjoy. At times I do wish it had better tooling and were built on a more-rigorous foundation.

I need to determine what to do next — spend more time with Clojure, or move on to Haskell. Also I need and want to spend significantly more time pairing with people here — which will at least partially entail working with a variety of other languages and platforms.

So it’s likely that I’ll take at least a brief break from Clojure. But I’ll return at some point.

Are printlns a sophisticated debugging technique? Nope. But some experienced programmers use them as a quick first resort (even when they’re willing and able to fire up a real debugger). ↩

Hands-on with Clojure day 4

Greg Hendershott — Tue, 14 Oct 2014 21:52:11 UT

Hands-on with Clojure day 4

2014-10-14 :: Clojure, Hacker School

Overview:

Spent time with Clojure Cookbook.
Started a cheat sheet for Racket ↔ Clojure.
Looked at split-with.
Started a port of wffi from Racket to Clojure.
Confusion: Aggregates and generics.

Clojure Cookbook

I discovered Clojure Cookbook, which describes itself as:

Clojure Cookbook doesn’t just teach you Clojure, it also shows you how to use the language and many of its common libraries. The most difficult part of mastering any language is knowing how to apply it, in an idiomatic way, to tasks that real software developers encounter every day. This is especially true of Clojure.

With code recipes that teach you how to use the language in a variety of domains, Clojure Cookbook goes beyond simply teaching Clojure syntax and semantics. It contains annotated example code with detailed analysis and explanation for hundreds of real programming tasks. You can read the book straight through to gain insights about Clojure, or use it as a reference to solve particular problems.

You can git clone the Clojure Cookbook project and work with it in Emacs. This is a nice experience.

You can read each section in an adoc-mode buffer, with cider-repl active. That way you can C-x C-e to see the result on the Emacs status bar (or if you prefer, paste or type directly in the the cider-repl buffer).

Plus you can bind a key like M-+ to an Elisp function that finds and advances to the next section.

I found most sections to be helpful. Not only do they show how to do X, they get into the rationale of when and why should use a certain Clojure approach vs. another. Very good stuff.

Racket ↔ Clojure Cheat Sheet

As I’ve learned Clojure equivalents for Racket functions, I’ve jotted them down in notes. The notes got long enough that I moved them into org-mode tables — resulting in this cheat sheet.

The first part covers, given Racket function X, what’s the exact or close equivalent Clojure function Y? The second part is for the other direction, from Clojure to Racket.

Again, these are my rough working notes. Don’t rely on it. I wouldn’t be shocked if some of it is subtly — or blatantly — wrong.

I found that, as is often the case with taking notes, the act of taking them makes it less-likely to need to refer to them.

Cheat sheets, `split-with`

Speaking of cheat sheets, on #clojure someone reminded me of the Clojure cheatsheet. This seems like something to keep open in a browser tab. Or, say, print and paste on the inside of my forehead.

I was on IRC asking if Clojure had some equivalent to Racket’s splitf-at, with an implementation faster than the conceptual one I was able to come up with:

(defn split
  "FIXME: This is the conceptual, inefficient implementation. Should
  re-implement like Racket's splitf-at."
  [pred coll]
  [(take-while pred coll)
   (drop-while pred coll)])

Someone pointed out split-with, which I could have found on the cheat sheet.

So that’s cool. However M-. on split-with shows an implementation that is… the conceptual one. I guess that could be a micro-project — write an implementation that doesn’t excessively traverse and create intermediate collections.

wffi for Clojure

A couple years ago, I was reading so much Amazon Web Services documentation that included what were essentially HTTP request and response templates. Such as the following, albeit much more complicated than this:

GET /users/{user}/?key={val}
Header: {value}
  ...

Eventually it dawned on me that it would awfully be nice if web service documentation like this could actually drive code generation. Both generate client wrappers to make requests to the service, and, generate API glue for the service’s server. That was the idea behind webapi-markdown. For example, you could take some service’s documentation, run it through a tool, and voila, an “SDK” in your favorite language.

So webapi-markdown is just a spec. The one implementation to-date is in Racket: wffi. (The name implying both “web FFI” and “stinky”.)

In searching for a somewhat more challenging project to do in Clojure, I figured this might fit the bill.

So I spent the last day working on it. It’s been slow going, but so far I’ve had a chance to learn:

How to use markdown-clj for parsing markdown. Although it returns an HTML string, which isn’t ideal, it’s easy enough to convert that using tagsoup into structured data. (I keep thinking of these as “Clojure’s equivalent of x-expressions”. Is this known as “hiccup”? Not sure.)
How to use Instaparse. I only got a couple hours into this, but so far it seems pretty spiffy. It was fun to see the acknowledgment in the README that Danny Yoo’s Ragg was the inspiration. It is definitely nice to use in the same way that Ragg is, and more.

And eventually this project will get into areas like:

Code-generation via Clojure macros.
Generating docs (via doc strings, I suppose).
Distributing a project on Clojars (if I take it that far).

So I think that’s a pretty good chunk of real-world stuff, which makes it worth pursuing this, even if there’s no value in releasing it per se.

The code is still pretty raw, and my grammar for Instaparse needs work. So I’m making commits, but not confident enough to push stuff to GitHub quite yet.

Aggregates and generics

One area I’m still confused by is Clojure’s variety of approaches to aggregate data types and polymorphism.

Coming from Racket, we have structs and generics. structs have compile-time defined fields. As a result:

They’re fast: A field is a direct offset.
They’re strict:
- You can’t accidentally add or omit a field.
- Structs can inherit fields from other structs, strictly.

Of course we also have dictionaries (hash tables and association lists) when we want runtime flexibility.

My current impression is that Clojurians would often use a map instead of a struct in many situations. I think?

Clojure does have things that seem to be in the same ballpark as structs. defrecord sounds like it is about structs. Is it sugar for maps? deftype is, erm… I don’t know. Clojure Cookbook has a quote from Chas Emerick:

Is your class modeling a domain value—thus benefiting from hash map-like functionality and semantics? Use defrecord.

Do you need to define mutable fields? Use deftype.

Hmm. That’s not what I would have guessed from the name, “type”. Plus, I thought mutability was strongly discouraged in Clojure.

The docs for defrecord and deftype seem a bit terse and opaque to me: The words make sense but aren’t crisply connecting to concepts I already know.

So, clearly I need to learn more in this space.

Requires are less often, but still, tricky

Although it’s much better, I still seem to get confused by requires. Good example is clj-tagsoup. I kept trying to :require pl.danieljanus.tagsoup.core. But it’s just pl.danieljanus.tagsoup. The classpath error message isn’t very helpful for me. If I go in ~/.m2 and look, I don’t see source files. I need to go up on GitHub and view the source. That’s how I eventually noticed it’s src/pl/danieljanus/tagsoup.clj. Not e.g. src/pl/danieljanus/tagsoup/core.clj.

To be fair, I suppose this is really about one library’s choice. And I’m new to Clojure. But I don’t seem to struggle with this in Racket. Especially when an installed package has Scribble-generated docs, the docs automatically show a (require foo) that, well, just works the first time.

Reflection and next steps

Notwithstanding my whinge in the previous section, I’m hitting tooling and cider speed bumps much less frequently. I felt like yesterday was the first time I could actually focus on the code, without distraction, for solid hours at a time. Although I still don’t feel like I’ve figured out an optimal Clojure workflow, it is easier for me to imagine such a thing being possible someday.

Next up, I plan to keep working on the wffi port, and see how far I can take that this remainder of this week.

Hands-on with Clojure day 3

Greg Hendershott — Fri, 10 Oct 2014 15:13:34 UT

Hands-on with Clojure day 3

2014-10-10 :: Clojure, Hacker School

Please see the usual disclaimers from my previous posts.

As I mentioned yesterday, my next toy project is to write wrappers libraries for the new Hacker News API. This seems like a good exercise because the REST API is very simple, and I have experience doing this sort of thing in Racket. In fact, I’ll do the same thing in both Racket and Clojure.

The result is clacker-news and racker-news. Trademark registration application is in-process.¹

Although I knew this would be a simple project, it turned out to be simpler than I expected:

The Hacker News API is just a half-dozen items, so that helps.
Both Racket and Clojure have easy ways to parse JSON into the relevant idiomatic representations. (As you might imagine, Racket is more listy and Clojure is more vectory, but they are pretty similar. Because JSON.)
I didn’t try to do anything fancy with error-handling. If there’s an HTTP-related exception, it’s going to bubble up to the user of my library. I think this is probably fine.
I didn’t try to do anything fancy with retries. This is probably less-fine. Given response like say 429 Too Many Requests, it would nice if my library would automatically retry some number of times (with an exponential delay back-off). But it doesn’t.

However, maybe it was unduly easy — because I overlooked something important in Clojure (or about making REST API requests in general). If so, feel free to hit the comments.

Redefinitions and the top-level

While doing some copy-pasta of the Racket source to Clojure, I accidentally ended up with two defns of the same function, get-user. The thing is, Clojure did not give me an error message. This really surprised me. In Racket, this would be a redefinition error.

I guess this is related to my observation in yesterday’s post about the need to use declare for forward references. If I understand correctly, this means that Clojure’s evaluation model is closer to a simple load: It is essentially equivalent to typing stuff at a top-level REPL prompt. Things are evaluated one s-expression read at a time.

So for example it’s perfectly fine to redefine something at the top-level in the REPL.

And, if you read things one s-expression at a time — as opposed to an entire file and/or namespace — you can’t know about something that isn’t defined yet. You need a hint like declare: “Chill, I’m going to supply it later.”

What I’m used to from Racket is an actual module system. As I understand it, the first unit of evaluation is a module. And the #lang business is a shorthand for module forms. In other words:

1
2
3

#lang racket
(define (id x)
  x)

is shorthand for:

1
2
3

(module racket
  (define (id x)
    x))

And in fact you will see older Racket files that use the module form explicitly like that (as well as newer #lang files that use module forms within, i.e. for nested modules).

Maybe I’m misunderstanding, and the business about modules is orthogonal to the business about redefinitions and forward references. Maybe that’s really about read-ing s-expressions one at a time like in a top-level REPL. In any case, I don’t love this aspect of Clojure.²

Next steps

I need an idea for a gradually more-complicated project to try, next.

Ideally it would exercise something special/strong about Clojure, such as the persistent immutable data structures, concurrency primitives, or so on.

(As a counter-example, I could explore macros in Clojure, but my expectation is that’s an area that might be disappointing compared to Racket. Instead I’d like to find something where I’m more likely to say, “Dang, I wish Racket did this.”)

If you have suggestions, let me know in the comments, and thanks in advance.

Kidding. ↩
Among Racketeers, a famous quote is, “the top-level is hopeless”. ↩

Hands-on with Clojure day 2

Greg Hendershott — Thu, 09 Oct 2014 14:58:46 UT

Hands-on with Clojure day 2

2014-10-09 :: Clojure, Hacker School

As with yesterday’s post, important disclaimers:

I’ve used Racket heavily but not Clojure.
Opinions expressed herein are not those of my employer, were I to have one.
If indignation lasts more than four hours, please seek medical attention.

Day 2 with Clojure was much more fun! I didn’t hit speed bumps with tooling and workflow, so frequently. I was able to focus mostly on the code itself, which was wonderful.

For a slightly more realistic task, I decided to make a really simple function that, given a URL, would make a GET request and return the value of the Server response header (if any), and a list of URLs found on the page that are for other servers. The idea being, you could crawl from some starting point and accumulate some data about web server technology.

To do this, I needed to understand how to:

Make HTTP requests.
Parse an HTTP response entity (the HTML document) in a way I could walk it to look for <a href="foo"> elements.
Miscellaneous other things.

Making HTTP `GET` requests

I discovered clj-http. Added it to projects.clj, and added (:require [clj-http.client :as client]) to my core.clj. I got errors in cider. It worked fine with lein repl at the command line. I did a cider-restart and eventually it started it working.

Take-away: I’m starting to learn how to supplicate the system reliably, with certain incantations. But I don’t yet have a good mental model for the statefulness of adding a lib to a project and using it, at least not with cider.

Using clj-http was a joy. Returns a nice map with key/value pairs I’d expect.

Parsing and walking HTML

Next, how to parse and walk the HTML in the :body?

In Racket we typically use what are called x-expressions to represent XML and (well-formed) HTML:

`(html ()
  (head ())
  (body ()
        (p ([attr "val"][attr2 "val2"])
           "Some text " amp " some more.")))

In other words this is an s-expression that follows a convention: There is a list. The first element is a symbol for the element tag name. The second element is an association-list of attributes. The zero or more remaining elements are either plain data like strings or numbers or symbols, or, other x-expressions.

I searched around and read about Enliven. That seemed like a heavier tool than I needed. On StackOverflow someone mentioned clj-tagsoup. That looked like exactly what I wanted — or at least was familiar with. To me, these look like x-expressions, but using vectors instead of lists, and using maps instead of association lists.

[:html {}
 [:head {}]
 [:body {}
  [:p {:attr "val", :attr2 "val2"}
  "Some text & some more."]]]

Wonderful.

Adding that to my project.clj and requiring it in my core.clj was again weird. From the GitHub README and from Clojars.org, I’m not always sure when to use an organization-id prefix, and if so, what to use. I must be confusing myself because this seems like it should be simpler. At this stage, I try things until it works. The only catch being, I might need to use cider-restart each time, otherwise I might not realize I actually did get it correct.

Anyway, once added and required properly, clj-tagsoup was also a joy to use.

Parsing URLs

Now that things were working, I noticed that the list of URLs included many links to the same server. There’s no point in crawling those — presumably the same web site is using the same web server. Instead I want to filter these to be only URLs for other sites, so we can go crawl those and discover what they return for Server:.

To do this, I want to split the URL into its components, and look at the scheme (a.k.a. protocol like “http”), hostname, and port, but disregard the rest. In Racket I would use string->url, which would return a url struct having fields for each of these. How to do this in Clojure?

I couldn’t find an answer in the Clojure docs. For a minute I thought about using a regular expression. But I’ve seen the regexp that Racket’s string->url uses. A correct regexp is non-trivial. Anyway this seems like a wheel I should not be reinventing.

I asked on #clojure IRC. Justin Smith and David Nolen quickly helped me out, showing me I could use (bean (java.net.URL. "http://www.google.com")), which returns a map, members of which include :protocol, :host, :port. Perfect.

Then I looked again at the map:

{:path "",
 :protocol "http",
 :ref nil,
 :content #<HttpInputStream sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@56e01e8a>,
 :authority "www.google.com",
 :file "",
 :port -1,
 :host "www.google.com",
 :class java.net.URL,
 :query nil,
 :defaultPort 80,
 :userInfo nil}

Hmm, what’s up with that part, :content #<HttpInputStream sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@56e01e8a>. That sure looks like an open connection. I don’t want to open a connection for all of these — just parse/split the URL string into its components.

It seems that bean calls all of the accessor members, and Java.Net.URL.getContent() has a side-effect:

1	openConnection().getContent()

So that’s not good. Back on IRC, just as I was typing, “I wonder if I can call the methods directly instead of using bean”, Justin Smith said the same. Turns out he’d already posted a benchmark of bean vs. directly calling select members, which is just what I needed.

Of course on my first attempt I managed to forget the trailing . in (java.net.URL. url) — but then figured that out.

Take-aways:

I needed to learn about Java interop earlier than I expected. The interop per se seems simple enough, and I’ll re-read that part of Joy of Clojure today.
I’m a bit worried I don’t yet know the Java library ecosystem. How will I know what’s available? But I guess library discoverability is a challenge for every language and language-learner. Maybe for Clojure the steps will be:
1. Search Clojure docs.
2. Search Clojars.
3. Search Java docs.
4. Ask on #clojure. Preferably as step 4 not 0. :)
Justin Smith suggested the javadoc-search-pane browser add-on for Chrome and Firefox as a great way to explore Java docs.
Justin Smith is incredibly helpful! Thanks!

Forward declarations

One thing I find slightly annoying in Clojure is the need to use declare, which isn’t necessary in Racket. If you like your source code to be “bottom up”, building up ingredients to a dramatic finale at the end of the source file, it’s not an issue. However, if you like to be “top-down”, starting with the main public product, and working down into the supporting cast, it can be awkward.

Code

(ns web-client.core
  (:require [clj-http.client :as client]
            [clojure.test :refer :all])
  (:use pl.danieljanus.tagsoup))

(defn- server
  [response]
  (get (:headers response) "Server"))

;; Grumble: Needing to use `declare` is annoying compared to Racket.
(declare body-elements
         hrefs)

(defn- urls
  [response pred]
  (let [entity (:body response)
        tree (parse-string entity)
        bodies (body-elements tree)]
    (hrefs bodies pred)))
    
(defn- body-elements
  [tree]
  (some (fn [node]
          (if (= :body (tag node))
            (children node)))
        (children tree)))

(defn- hrefs
  "Given a collection of body elements, and a predicate, return all of
  the link URLs on the page satisfying the predicate. For example the
  predicate might be if the URLs are for another hostname, i.e. a link
  to some other web server. "
  [xs pred]
  (filter pred
          (flatten
           (map (fn [x]
                  (if (coll? x)
                    (if (= :a (tag x))
                      (:href (attributes x))
                      (hrefs (children x) pred))))
                xs))))

;; TODO: Move this test elsewhere.
(is (= (find-hrefs [[:p {} "par"]
                    [:a {:href "http://hi.com"} "hi"]
                    [:p {}
                     "par"
                     [:a {:href "http://there.com"} "there"]
                     "par"]])
       ["http://hi.com"
        "http://there.com"]))

;; Grumble: Needing to use `declare` is annoying compared to Racket.
(declare valid-url? scheme+host+port)

(defn- server-equal?
  "Predicate for two URLs having 'the same server', as determined by
  having the same scheme, host, and port.

  If either URL is malformed (according to java.net.URL) then this
  returns t. That's because the intended use of this is by
  `get-server-and-links'. Of course that means the name
  `server-equal?` isn't quite right, and probably all this should be
  refactored. TO-DO."
  [a b]
  (or (not (valid-url? a))
      (not (valid-url? b))
      (= (scheme+host+port a)
         (scheme+host+port b))))

(defn- valid-url?
  [url]
  ;; Note: java.net.URL can throw an exception if the string is not a
  ;; well-formed URL. In that case, return some default map.
  (try 
    (java.net.URL. url)
    (catch java.net.MalformedURLException e nil)))

(is (valid-url? "http://www.google.com"))
(is (not (valid-url? "#anchor")))

(defn- scheme+host+port
  "Given a URL, return a map of its scheme, host, and port."
  [url]
  (let [loc (java.net.URL. url)]
    {:scheme (.getProtocol loc)
     :host (.getHost loc)
     :port (.getPort loc)}))

;; TODO: Move this test elsewhere
(is (= (scheme+host+port "http://www.google.com")
       (scheme+host+port "http://www.google.com/path/to/foo?q=0;p=1")))
(is (not (= (scheme+host+port "http://www.google.com")
            (scheme+host+port "http://www.giggle.com"))))
(is (not (= (scheme+host+port "http://www.google.com")
            (scheme+host+port "https://www.goggle.com"))))
(is (not (= (scheme+host+port "http://www.google.com:80")
            (scheme+host+port "http://www.goggle.com:8080"))))

;; TODO: Move this test elsewhere
(is (= (scheme+host+port "http://www.google.com")
       {:scheme "http", :host "www.google.com", :port -1}))
(is (= (scheme+host+port "http://www.google.com/path/to/foo?q=0;p=1")
       {:scheme "http", :host "www.google.com", :port -1}))

(defn get-server-and-links
  "Given a URL, visit it and return a map with the value of the Server
  response header (if any), and a collection of URLs for other hosts
  to which it links (if any).

  In other words, the intended use is that you'd crawl and accumulate
  data about web server technology for some subset of teh interwebs."
  [url]
  (let [response (client/get url)]
    {:server (server response)
     :urls (urls response (complement (partial server-equal? url)))}))

(comment
  (println (map get-server-and-links '("http://www.google.com"
                                       "http://clojure.org"
                                       "http://www.racket-lang.org"))))

Next Steps

UPDATE: Oops, I posted without this part.

What next? Given that:

I’m a roll with HTTP stuff in Clojure.
I’ve done wrappers for web services like AWS and GAPI in Racket.
Hacker News has a new, simple REST API.

I think I might try writing wrappers libraries for Hacker News in both Clojure and Racket. As a plus, I can name them clacker-news and racker-news. I mean, maybe that’s a plus.

Thursdays are alumni days at Hacker School, and if I understand correctly there are lightning talks (maybe this afternoon?). So maybe I’ll have less time to work on stuff today. But I can carry it over to tomorrow, as well.

Posts tagged "Hacker-School"

Blogging catch-up

So what have I been up to?

Hacker school week 6

Hands-on with Haskell week 1

Rewiring brain from Lisp to Haskell

Sections

Commas

Other tiny things

Nice things

My first hands-on with Cabal

Porting wffi to Haskell

Surprisingly easy

Applicable symbols

Editorializing

Postscript

Hacker School day 15

Why macros?

Hands-on with Clojure day 5

Clojure port of wffi

split-with and lazy seqs

Load vs. modules

Conclusions and next steps

Hands-on with Clojure day 4

Clojure Cookbook

Racket ↔ Clojure Cheat Sheet

Cheat sheets, split-with

wffi for Clojure

Aggregates and generics

Requires are less often, but still, tricky

Reflection and next steps

Hands-on with Clojure day 3

Redefinitions and the top-level

Next steps

Hands-on with Clojure day 2

Making HTTP GET requests

Parsing and walking HTML

Parsing URLs

Forward declarations

Code

Next Steps

`split-with` and lazy seqs

Cheat sheets, `split-with`

Making HTTP `GET` requests