golang package cappedqueue
To quote the README.rst file:
cappedqueue
cappedqueue is a simple package to make sure you can submit to a queue. Losing items is on purpose so that you can rely on not accidentally blocking or filling up your memory
see the
cappedqueue_test.gofile for details on the usage…/serverhorror
The Importance of API “Transports”
SQL Databases are dead
There I said it. I jumped the NoSQL waggon and I refuse to use anything that is remotely uncool and old.
Of course the above is a lie. Well kind of. In reality I do think that the interface by which you communicate with a SQL database is dead (or rather should be, but so should FTP — another story — still there are a few people left using it).
I’m not really talking about the query language itself, I’m talking about the “transport” (lacking a better word).
So is (nearly) any other protocol
I’ll leave the kool new aid road now and talk about the transport stuff a bit.
My problem with API interfaces currently is that the usage of these interfaces is quite annoying. Let’s look at the products that people use all over the web.
Say you go to Facebook or Google and what to use their API to do something with the social networks represented by the data. On the API page of Facebook you’ll find some source package that implements a transport protocol, a domain specific language and to complicate things a bit further also a certain data representation.
Reality is that right now to use some service on the Internet there is only a single transport protocol. HTTP, maybe add SSL on top of that but the transport is HTTP. I’m not saying that the HTTP protocol is the ultimate protocol. It’s shortcomings are quite well known.
The point is that you can solve a whole set of problems all in one go and be done with it.
Back to the SQL Example
So you have this cool new project and decided to use some relational database system. Now you also decide to use a framework that relies on interactions with other systems being non-blocking. So you run around the web and look for some non-blocking driver for you RDBMS of choice.
Then you run around the web and look for some non-blocking driver for you storage part, then you run around the web…(you get the idea).
Now if all your satellite systems would be exposed oder HTTP you’d…run around the web and look for a non-blocking HTTP client and be done with this category of problems.
Just don’t use HTTP…please!
Please, please, please…don’t just use HTTP because I used it as an example here. A unified (or generalized) transport really needs some serious thinking.
Of course you can just run away now and scream what kind of idiot I am, but the next time you run around the web and and look for a driver because the transport of services isn’t generalized enough to be reusable.
Also please don’t confuse the transport with the represenation — which should be considered equally important and would solve another stack of problems.
And if you go that last route, that still doesn’t say that should be only a single language per domain, having a single language per domain would be not that good without wasting further thoughts about those last two statements.
/serverhorror
What is big data?
It’s actually quite simple: You have big data whenever a single host isn’t enough to either store or process your data.
What does it mean?
Suppose you have a Postgresql Database and you run into scaling problems. There’s a choice now, either get better hardware so that you can continue to work on a single host or split the database to span multiple hosts.
Suppose you have a file store and all disks are full. You can either buy larger disks or use some distributed storage system where you just add hosts to expand the total storage capacity.
In both cases you are dealing with big data.
Big data (for me) isn’t anything that says X MB of data. It’s simply the case when you need decide to use a distributed system to handle your data.
Awk min/max/avg
#!/usr/bin/awk -f
BEGIN
{
minimum=0;
maximum=0;
sum=0;
}
{
if($3>maximum)
{
maximum=$3;
}
if($3<minimum)
{
minimum=$3;
}
sum+=$3;
}
END
{
print "Average = ",sum/NR;
print "Max = ",maximum;
print "Min = ",minimum;
}
NON-RFC953 Hostname Service
- /etc/xinetd.d/hostname
service hostname {
disable = no
type = UNLISTED
id = hostname
port = 101
socket_type = stream
protocol = tcp
user = nobody
wait = no
server = /bin/hostname
server_args = -f
}
Monitoring Thoughts
Monitoring Thoughts
How would you scale monitoring and how would you ensure that with hundreds of thousands of events per minute you’ll still get the important ones?
A lot of stuff is missing here. This is merely a note how I think a scaling architecture for monitoring should look like. Also one should be able to do math on the events!
On Agents
RULE: events generated by Agents are stateless
- Run a monitoring agent on each node!
- Each agent performs a number of tasksThese are specifically called tasks since those are not necessarily checks. Also I associate checks with nagios checks. It’s not what we want to do!
A task does one thing, and one thing only:
- do not create tasks that are what NimSoft does (CMD – CPU/Memory/Disk)
- Each tasks generates an event
- Everything is an event!
- A successful task just a taks that ran without (programmatical errors)
- A failed task is something where a programmatical error occured!
- Create a JSON String from the event
- Submit the JSON string to some messaging middleware (preferrably RabbitMQ)
On Middleware
- Messagesmust be persistent
- It is safe to restart the server!
- What are just messages to the middleware are the guts of the system.Those are the events generated by agents
On Servers
There are 2 kinds of servers:
- PersistenceServersThese run somewhere in a rack. They will grab one event after another from a queue and store them in a safe place for later reference.
Once a Persistence Server grabbed an event from the queue it is no longer visible to other servers. Each event will reside on and exatly on Persistence Server.
- NotificationServersThese run either on physical serves in a rack and just grab one notification at a time from the messaging middleware.
- All notification server can retrieve all events.
- Notification Servers can subscribe to a certain subset of topics.
- There may be a lot of servers. We don’t want our monitoring failing
On Persistence Servers
- Subscribe to the global queue
- Start grabbing events
- Store the event on diskWhat exactly storing means is yet to be determined!
- Start over again
On Notification Servers
- Subscribe to the notification queue or a topic queue
- Start grabbing events
- Display the eventWhat exactly displaying means is yet to be determined!
- Start over again
