Skip to content

Insight and analysis of technology and business strategy

Minimal Twitter to Google Pub/Sub example with Scala

Recently I was looking for a simple Twitter to Pub/Sub streaming pipeline and ended up with own implementation in Scala. I tried to make it as compact as possible. So I chose the dispatch and Google Pub/Sub client libraries for Java. You should have a Google Cloud Platform service account key and Twitter API consumer key and tokens ready to start. 1. Create publisher: [code language="scala"] val publisher = Publisher. newBuilder(TopicName.of("projectId", "topic")). setCredentialsProvider(FixedCredentialsProvider.create( GoogleCredentials.fromStream(new FileInputStream("key.json")))). build()[/code] 2. Get stream of statuses/sample messages and publish them! [code language="scala"] Http.default( url("") <@ ( new ConsumerKey("consumerKey", "consumerSecret"), new RequestToken("accessToken", "accessTokenSecret")) > { tweet => publisher.publish(PubsubMessage.newBuilder.setData(ByteString.copyFromUtf8(tweet)).build()) })[/code] As improvements you may want to configure BatchingSettings settings for Publisher and add various exception handlers. You can find full source code here.

Top Categories

  • There are no suggestions because the search field is empty.

Tell us how we can help!