Technical Track Dba

Minimal Twitter to Google Pub/Sub example with Scala

1 min read

Dec 13, 2017

Recently I was looking for a simple Twitter to Pub/Sub streaming pipeline and ended up with own implementation in Scala. I tried to make it as compact as possible. So I chose the dispatch and Google Pub/Sub client libraries for Java. You should have a Google Cloud Platform service account key and Twitter API consumer key and tokens ready to start. 1. Create publisher: [code language="scala"] val publisher = Publisher. newBuilder(TopicName.of("projectId", "topic")). setCredentialsProvider(FixedCredentialsProvider.create( GoogleCredentials.fromStream(new FileInputStream("key.json")))). build()[/code] 2. Get stream of statuses/sample messages and publish them! [code language="scala"] Http.default( url("https://stream.twitter.com/1.1/statuses/sample.json") <@ ( new ConsumerKey("consumerKey", "consumerSecret"), new RequestToken("accessToken", "accessTokenSecret")) > as.stream.Lines { tweet => publisher.publish(PubsubMessage.newBuilder.setData(ByteString.copyFromUtf8(tweet)).build()) })[/code] As improvements you may want to configure BatchingSettings settings for Publisher and add various exception handlers. You can find full source code here.

← Replicating in Google Cloud SQL using Tungsten

How to move or clone a Microsoft Azure virtual machine? →

Get Email Notifications

No Comments Yet

Let us know what you think

Minimal Twitter to Google Pub/Sub example with Scala

Share this

You May Also Like

Best Practice Recommendations for Windows Hosting SQL Server on VMware

Oracle Database: Query to List All Statistics Tables

Exposing MyRocks Internals Via System Variables: Part 7, Use Case Considerations

Get Email Notifications

No Comments Yet