Process 30–50 million tasks per day; max throughput of 2k TPS (tasks per second). Tasks should not be duplicated (executed exactly once). Some tasks need to scheduled for later.
When we first introduced celery to our existing django application, we were at less than 1/100th of the traffic we have now. Life was simple-
some_task.delay(), it returns immediately and executes asynchronously
As we scaled more than 100x, over the next 2 years, life become complicated.
One way to write software is to write some code, then test it manually, make some more changes, test again, until you’re satisfied. Then, maybe, you write tests.
In TDD, on the other hand, first you write a test case (that fails), then write code to make it pass, make some more changes while making sure the test pass. Once you’re satisfied you run the whole test suite to make sure nothing else has broken.
There are quite a few rules/principles that get thrown around in the software world. Some that come to mind are SOLID principles, Design Patterns, Do one thing and do it well, etc. I totally stand by all of them and believe our world would be a much better place if these principles are followed more.
The problem, however, is that those principles are quite abstract and at a very high level. During my code reviews, engineering discussions and writing of code, its tough to apply them directly. You wouldn’t know when you’re violating any/some of them.
I thought of compiling…
Have you ever asked yourself questions to which you can’t find the answer, no matter how much time you spend on it, or reason about it. I know I have.
“Should I break-up with my girlfriend?”
“Should I quit my job and start a company?”
“Should I shut down my company and take a job?”
These questions are hard, we know that. But why are they hard?
I argue they are hard not because it’s tough to answer them, but because there is no right answer.
Surprisingly, this applies to not only to life’s philosophical questions but also to well-defined…
I recently went through a job change and would like to post my experience. My post got a lot of people interested on reddit.
TL;DR: Lot of companies do not focus on good engineering. Prepare. The algorithm rounds were a hit-or-miss for me.
Recently, I had the misfortune to look for a rental house. Besides parsing through multiple properties on property rental portals, I had to contact agents. For every property that I liked (and had a chance to get):
I had to copy the agent’s phone number from the computer screen to my phone, wait for WhatsApp to sync contacts, mentally note the listing name from the computer, and only then send a message from my phone.
And, I would have to do it 79 times:
By good engineering I mean following good engineering practices and writing maintainable code. It includes, but is not limited to doing Test Driven Development, writing tests, using the CI and CD, implementing a design pattern, religiously doing code reviews.
By getting shit done I mean doing whatever little is needed to implement the feature. Just mark the jira ticket to done without doing any of the things mentioned above. This will invariably lead to bugs.
Some incidents at work prompted me to post on reddit: Writing high quality maintainable code v/s getting shit done? I felt like I’m forever stuck…
I was really excited when I realized I can break down my app’s code into smaller intuitive components, rather than separating the “business logic” from the framework code: Custom Views as Components. I told everyone at work and developers outside work about my newfound treasure. I felt this could actually be a viable architecture, right up there with MVVM and MVI.
But then it hit me: How can one write unit tests for components?
In order to be a viable architecture, it had to be unit testable.
After some soul searching (read: googling), I became confident of what I my…
Programs that write programs! Or (more formally), a programming technique in which computer programs have the ability to treat programs as their data.
While working on my startup, I came across an interesting problem. In our web-app, we needed to create
Portfolios of our artists (actors, models, photographers, etc.). We were constantly adding new categories (singer, dancer, etc) as well as more and more attributes (experience_level, has_agent, languages) to existing categories.
The solution to this problem is one of my fondest memories in writing code. I believe it’s for moments like this that engineers write software.
How will you organize your android app’s 10k+ lines of code written by 10+ people? In a couple of months, before you know it, it’ll cross a 100k lines. Wouldn’t it be great if when a new feature requirement comes, you create a new class and start working on it, rather than worrying about what other things might break if you edit this 5k LoC Activity/Presenter class?
Create a new component!
Components let you split the UI into independent, reusable pieces, and think about each piece in isolation.
- react js
A typical example:
In the example, each of the…