Design a push queue using Amazon SQS and SNS, how to?
|February 18, 2012||Posted by forumadmin under TechQns||
- App #1 insert messages to SQS
- A .Net worker role polls the queue and if message found sent it to App #2 to process, wait for App #2 to finish and re-start polling the queue
- App #2 process the message which is a long and heavy task
As App #2 could take quite long to process the messages and I cannot forecast how many messages are sent simultaneously from App #1, the queue system guarantees me that App #2 doesn’t run out of resources and the system can scale quite easily. But there’s a problem I’d like to solve: I would prefer not to have a machine to run the worker role (now on Azure) just to poll a queue (everything is hosted elsewhere where worker role is not an option). Moreover polling won’t ever be so responsive as push because of pauses between polls.
Switching from poll to push sounds the right path but I need I need to keep the guarantee that even if 1k messages are sent from App #1 in one second, App #2 process them one by one and is not hit 1k times per second.
I was planning a design where App #1 publish to SNS Topic whose subscribers are SQS and App #2. App #2 checks the SQS queue and if it is empty it quits, if not process the messages one by one and then quits. But how can I code App #2 (right now a .Net web/web service) so that if it is already processing a message and receive a notification from SNS it does nothing and quits (if not multiple processing would be run).
Any suggestions how to design this? I read this blog post but I do not know how to avoid the processing app to process multiple messages simultaneously.
|Asked By – Pietro||Read Answers|