Technical Track DBA

Minimal Twitter to Google Pub/Sub example with Scala

1 min read

Dec 13, 2017 12:00:00 AM

Recently I was looking for a simple Twitter to Pub/Sub streaming pipeline and ended up with own implementation in Scala. I tried to make it as compact as possible. So I chose the dispatch and Google Pub/Sub client libraries for Java. You should have a Google Cloud Platform service account key and Twitter API consumer key and tokens ready to start. 1. Create publisher: [code language="scala"] val publisher = Publisher. newBuilder(TopicName.of("projectId", "topic")). setCredentialsProvider(FixedCredentialsProvider.create( GoogleCredentials.fromStream(new FileInputStream("key.json")))). build()[/code] 2. Get stream of statuses/sample messages and publish them! [code language="scala"] Http.default( url("https://stream.twitter.com/1.1/statuses/sample.json") <@ ( new ConsumerKey("consumerKey", "consumerSecret"), new RequestToken("accessToken", "accessTokenSecret")) > as.stream.Lines { tweet => publisher.publish(PubsubMessage.newBuilder.setData(ByteString.copyFromUtf8(tweet)).build()) })[/code] As improvements you may want to configure BatchingSettings settings for Publisher and add various exception handlers. You can find full source code here.

On this page

Ready to unlock value from your data?

With Pythian, you can accomplish your data transformation goals and more.

Speak with Pythian consultants now →

What We Do

Minimal Twitter to Google Pub/Sub example with Scala

Share this

Share this

More resources

Apache beam pipelines with Scala: part 1 - template

Apache beam pipelines With Scala: Part 2 - Side Input

Building a custom routing NiFi processor with Scala

Ready to unlock value from your data?