Business Days: How i made 7k connections in 2 hours

September 18, 2016

business-days-how-i-made-7k-connections-in-2-hours  Facebook  Twitter  Google +  
  thumb.php?f=6IlrwIrOrg_5QizlbxzzbaMz9e8lt_w_CEi542DxF-e9fFq_YngBD5EB8cviaDbhx5QxSUHC9nesDRxOs7zYig..&w=360

 I realized something in this year, marketing is more precious than development. so kinda quit the freelancing and tried to reach people in numerous ways. look at the homeless people, they're living in crap but got access to people, not selling anything but even they can make money just by asking. Engineering & development can be done in shallow waters, when time comes to marketing you have to dive inside oceans with full of sharks. And you have to make sure you got proper weapons before sailing.

19_36_34_183863734.jpg2016-09-17_15-58-21.jpg

 
 First Attempt
 Few weeks ago, spared my dentist money and used them to advertise one of our ongoing products. first address was twitter. Designed a banner, launched a campaign and started to getting results in next couple hours. three days later their dashboard shows over 15k impressions and 1k interactions. Thing is i didn't see these interactions in server side, even a single click!
 
 Stopped the campaign to prevent any more damage and switched to facebook. was thinking at least they're strictly disallows the bot accounts by verifying people with phone number & national identity etc. This time got 500~ impressions and around 100 clicks but all of them bounced back without any sign up's.
 
 I think problem was fb's audience management. Around 30 people "liked" the advertising post, i checked each of them, their business titles was completely unrelated with mine.
 
 a similar story happened at adwords, i stopped all campaigns at the total $420~ damage and decided to go oldschool. like 2000's develop an application, send bulk mails, post entries to forums.
 

 Google's limitations
 I wasn't know where i start, just typed what i want to search engine. then written few lines of php snippet by using previously posted spider class to automate it but google started to block my requests by asking captcha created a solution to solve captcha's myself by taking 10 seconds of break on script execution but still results stalled on 150~ entries.
 
 Then realized google says "Wohoho i got billions of results" but only returning 50 pages of them. try it right now, search something and go straight to latest result page. it's mostly same in all search engines.
 

 Final Solution
 Next day developed a basic browser by using webkit based CEF library It's sending page source to crawler script while i'm browsing pages and injecting a portion of javascript to automate clicks. If we can define php based robots as the "terminator" this one is definitely the robocop. you can decide who to kill, who to save. even solve captcha's yourself.
 
 after 2 hours, collected 7500k entries just by solving captchas on each 15 minutes, removed duplicates by using MSP utility and got link of 6900 unique digital agency websites in total. another script which i created years ago, connected to all links one by one and collected names and phone numbers from these websites. If interested, compiled binary of the project available on end of the entry.
 

 E-mail marketing is kinda dead with new global spam policies, waste of time. Calling each of them manually and sparing few minutes can give more reliable results.
 
 At least hearing few "no, not interested, be gone, fck you" from various voices sharpens the people's own ego. 

 Resources
 RoboBrowser.zip
 Mass String Processor
 class.spider.php

  

New gadget: wireless sensor station

August 08, 2016

new-gadget-wireless-sensor-station  Facebook  Twitter  Google +  
  thumb.php?f=eVMO6kSoPVPWrw7G6wK480CwC_Yp1t4GO7PkIDHA1evd_TTVZVjmLjsQPybYtk1N&w=360

 After the spending months to commercial projects, taken few days of break and created a new sensor node for home network. it's supposed to be a weekend project but i have to put rest of the new week to make it completely functional.

 
 One month ago i mentioned about purchased 2x pair of raspberry pi from an used goods store. still using first one as headless torrent client and repurposed second one as a sensor station at these days. sounds like overkill for such a simple task but reliability is the key while developing a security gadget. consistent readers possibly know, i was created it's arduino+ethernet version at past year but didn't mention about ruined nights which caused by false alarms.
 
 Preparing parts
 There is not too much thing to highlight, it's exactly same setup which mentioned in this entry got raspi with wifi dongle in this setup instead arduino. implemented mq5 gas sensor to both of them as an extra. my though was reading analog inputs by using MCP3008 adc converter but that caused a file descriptor leak which is very common in popular linux distributions. so i got laying around multiwii flight controller from a crashed quadcopter, i attached sensors to it and fetched data by using serial port. that took my 30 minutes instead spending hours maybe days to research what caused the memory leak in previous attempt.
 

 Software setup
 It's powered by node.js like other pair of pi and uses same libraries which i developed for torrent client project. raspberry's primary functionality is video transmission in that case. it's own video capturing tool raspivid can record high quality videos by utilizing the onboard h264 hardware. that's why ffmpeg and similar solutions stuck in 320x240.

 After more research found a guy's blog who forked the raspivid's source and implemented a live streaming feature which is exactly what i need. installed it for future use but recording is okay for now, i created a quick script to record 24hrs of video separated by hours and delete them everyday. another script creates a web daemon in a spesific port and captures photos and handles basic commands such as shutdown & restart etc.
 third script fetches data from arduino (multiwii) and pushes them to home automation server. sources are available in rest of entry.
 
 Functionality
 I planted it to living room which sees the balcony directly, now it's non-stop recording hd video, measuring temperature, light level and lpg & co2 gas levels and detects the human movements. as mentioned in first lines on entry, previous arduino based sensor was waking me up with false alarms. now using this device's data as cross-validator for incoming data. for example, if gas levels higher than 500 and other device's level is close to 500, my automation software (judith) launching the gas alarm.

 Same method applies to pir sensor, if you need to go living room, you have to pass from hallway first and hallway also got a pir sensor, machine records this activity and does nothing if movement detected in this order. if suddenly an activity detected in living room, that means someone break in from window or balcony, that's a reason to launch a fully audible alarm. as mentioned before, nobody going to break into my house and steal something but increasing the security perimeter increases the safety feeling and you don't have to worry tick or noises on middle of the night. commercial pir sensor sets provides same functionality but you have to arm them before goto bed, disarm before wake up for pee, arm again before sleep. almost no logic in there and completely depends on human commands.


 by the way temperature was approx 35 C while i recording the video, noticed the pir sensors cannot detect humans with higher temperatures, i was literally sweating and both of sensors didn't see me. validated that by approching to building's automated hallway lights. same condition applies while wearing raincoat. so we can say pir sensors not too reliable in that case. alternative hardware might be a doppler radar (the sensors which installed on automated glass doors on the public malls) but they're expensive and view distance is very limited.
 

 Cost
 Already got all components in my trash dump, so cost is almost free to me. component list follows:
 - Raspberry pi model B with official 8mp camera
 - 1x mq5 lpg gas sensor
 - DS18B20 temperature sensor
 - 1x light dependent resistor
 - multiwii flight controller as equivalent to arduino nano
 - 12v to 5v/1.2amps step-down converter (was sony phone's official car charger)
 - audio amplifier which salvaged from broken beats headphones
 - 1x speaker which salvaged from a dead notebook
 - TP-Link TL-WN722n wifi dongle
 - 1x voltage divider module
 
 plus spent exact 7 days to development but that's absolutely worth it. if interested, complete project sources available in this github repository.
 
 watch on youtube if having troubles with flash

  

Raspberry pi as headless torrent box

June 23, 2016

raspberry-pi-as-wireless-torrent-box  Facebook  Twitter  Google +  

 Raspberry pi still an expensive toy to me with it's $35 price. few years ago it was advertised as "credit card sized $5 computer" but we didn't see that number at anywhere.

DSCF7006.jpg
19_18_52_100969867.jpg20_22_27_102877359.jpg

 
 Few weeks ago i found an used wholesale store at a local market and purchased 2x units of camera+sdcard+board bundle for an acceptable price. my thought was installing it as a wireless security camera but that's didn't work due to it's low hardware specs. so i decided to build a torrent box with it.
 
 Low latency HD stream possible with ffmpeg binaries but you can't go higher than 512x512@15 fps, my current analog cameras already capable to deliver 512x480@60fps with 5ghz wireless bridge. problem is related with it's architecture, specs says 700mhz but actually it's identical to an 266mhz pentium II processor when we count the cpu cycles. so we can say it's an expensive arduino with usb host and a bit extra processing power. ethernet port does not count because ethernet chip is also an usb device which attached to board's internal hub.
 
 Anyway, sometimes i was noticing the free wifi of a nearby public mall. installed transmission torrent client and configured my portable hdd as file storage to take advantage of it. already got a high gain wifi adapter to get signal from 500 mt distance, it's driver installation was painful but i figured out at last. also there is another torrent client available for pi called deluge but transmission is more stable and also supports the remote sessions from other platforms instead single web dashboard.
 
 Another problem!
 this time torrent daemon starting before hdd automount then terminating with error. I decided to follow the way i know and installed node.js and develop a script which can handle order of system commands. sending system vitals to desktop pc is a plus.

 Currently there is only node's early beta binaries available on web, you have to build it from sources if you're really need it. no joke, compilation took exact 3 days. finally installed node 4.x and created a library for measuring board vitals. then created another script to pump data to my automation server.
 
 Final boss battle
 My active node version was 5.x at the desktop and wasn't aware about memory leaks of older releases. after one week of headless execution, app's cycle latency increased day by day and finally crashed due to memory hog. so i revised my code and installed nightly 6.x version of node. compilation took another 3 days and finally we're won the game.
 
 Setup
 - Raspberry pi and sdhc card
 - High gain wifi dongle
 - Portable USB hdd
 - Li-ion backup battery for cellphone
 
 We're spent 1 week and $15 per unit but still result is satisfying if you don't exceed 2-3 active torrents. In my opinion spend few bucks more and purchase a more bit professional board such as beaglebone to save time or buy expendable asian variants which you can find billions of them by prefixing "pi" with other fruit names while search.
 
 sources available on github:
 libraspi.js
 app.js


  

AC line detection with two components

May 26, 2016

ac-line-detection-with-two-components  Facebook  Twitter  Google +  
  thumb.php?f=i5Km_s9Hltb2uVM4-ow27O4NezN3OpQdW0zygFQy3jABkUkFz8vQ3xtnky5XRHF0QudrORE7HyLsQFeBMbcEoA..&w=360

 few months ago my 4yrs old ups died due to water exposure while working at balcony. it was very functional thing, can handle 2kva load which powering electronics of whole home, provides detailed measurements about AC line and battery status etc. even you can sense the whole district's power load by observing the ac line graph.

ldr.jpg
IMG_20160225_145144.jpgpen.jpg

 
 Tried to buy same device but no luck, even manufacturer company not exists anymore. so i purchased an used ups with $15 and upgraded it to handle more capacity. details available in this instructables entry.
 
 Requirement
 It's working fine but my software customizations become obsolete with this ancient hardware. decided to measure electricity myself by using third party arduino modules but none of them simply telling is electricity exists or not. i was about to connect 5v cellphone charger output to arduino's analog pin then got the idea while testing it by using my screwdriver.
 
 Solution
 Most screwdrivers got NE-2 neon lamps to test mains voltage which beaming a barely visible light when connected to AC voltage. i removed the lamp, attached a photoresistor (aka LDR) and wrapped them with black tape to prevent any external light source.

 tada! i have a AC measurement module with analog output. connected it to my sensor box and did minor modifications on software stack. now my home automation software can detect power interruptions and take actions such as turn off appliances to save battery or go hibernate to prevent data loss.
 
 current setup not measuring remanining battery but it's fixed, current capacity is exact one hour, i configured the instructions to automatically take action after 30 minutes.
 
 Total cost
 + 1x screwdriver with test feature
 + 1x photoresistor
 + tape and bunch of cables
 + male electrical plug
 + 1 business day of software development.

  

Gadget upgrade: Relay Box

May 05, 2016

gadget-upgrade-relay-box  Facebook  Twitter  Google +  
  thumb.php?f=A25pe0Bohx0pqXAcdNQ3d6IH3AIdzLfEReum6V0gQYUMXMmIdJfnilnEzNdAttXV&w=360

 At the first days of 2016, mentioned about my new relay box in that entry it contains a ssr relay module which can handle 220v voltage with 2amp current. the motivation of building a new box instead using existing one was emf problem, each cycle of relay causing electromagnetic interference and resetting the onboard microcontroller. sounds like not big deal but it's serious when your computer turning on the jukebox at highest audio level and playing music about 3 days while you're out of city. true story.

 
 We're eliminated this issue with a new box but problems are never ending. this time messing with extreme heat. i put an extra relay which can handle upto 220v/10amp for solder and stove, everything was going good until relay melts the plexiglass structure of box.
 
 Solution
 I got a small aluminium case laying around there which in exact size of the box, i mounted it by using 10mm screws with one finger distance. plus integrated a micro sized 12v fan for box cooling. such a minor functionality but result is looking like fusion generator biggrin.gif
 
 while it's open, i also implemented an AC capacitor to terminals which gathered from a dead hair dryer and guess what, that's completely solved the emf problem. at least it's stable since 5 months.
 
 Usage
 So what it does now, it's connected to my home automation software. previously mentioned about air quality sensor, it's measuring the air quality with various sensors. today it's the same device, just redesigned the casing and added connection heads for external PIR sensor same software analyzes sensor values from this device and turning on/off connected appliances based on levels. for example, i don't smoke cigarette even i hate it. if somehow a guest smokes at another room, particle levels in environment increases about 10% and software turns on the air ventilator fan at the wall. same scenerio applies to LPG, extreme deodorant usage and even fart biggrin.gif
 
 another example is air conditioning, i survived from whole winter with my small stove. it's enough for 5.5m2 room in our climate area. sofware determining first if there is someone exists in room by pir sensors then launching the stove for 20 minutes if temperature is below 15c.
 
 whole workstation also connected to a portable ups, i created a battery monitoring system afterwards to prevent take any automatic actions. cheap ups systems does not have a usage limiter, we don't want to fry our batteries if got electiricity interruption on stove usage.
 
 Cost
 another motivation was production price. maybe not at 2002 but similar devices available on ebay today starting from $100. my total spent resources is:
 
 1x 10A ssr relay: $10
 1x 2A ssr relay module (4 relays): $15
 5x female electricity outlet: $5
 Plexi casing already available but let's say $2
 
 Total spent: $32 + 1week

  

Indoor gardening ep2: Monitoring vitals with esp8266

April 17, 2016

indoor-gardening-ep2-monitoring-vitals-with-esp8266  Facebook  Twitter  Google +  

 usually companies creating understandable terms for each new product category like "blue energy balls" or "flapping little bee" the internet of things is one of them. wikipedia says it's the term for communicating embedded devices each other via wide area networks. actually that's cool way to describe microcontroller with tcp stack which is already exists in market since decades.

 
 in previous air freshener entry, i mentioned about my implementation attempt of esp8266 based esp-01 board to a basic security system. it wasn't worked but i figure out the problem later and started to build a remote telemetry probe for my next set of crops.
 
 esp-01 is controllable via serial AT commands. even you can make http requests like multi-purpose gprs modems. first of all, i was stuck at the network scanning part. that's related with power consumption. it's hitting to 200ma while scanning & connecting to wifi networks which is close to supply limit of arduino board. i fed it with an external +3.3v power supply and solved this one.
 
 second problem is uart noise, obviously that board got the firmware from early esp days and not accepting the bitrate set command. i applied a cloud update command which is seen in reference document and completely bricked the device. i was assuming a firmware update will bring these advanced commands and fix that issue by minor a tweak. reviving the module is another story, i mentioned at this instructables entry.
 
 first tests was based on arduino mini pro decided to go with uno r3 to accelerate software development part after few write cycles, plug-in & plug-out on each upload is literally a torture. got a stable sketch after 4-5 revision.
 
 currently it's sending me the outdoor temperature, soil humidity and outdoor light level. same as previous setup, my personal assistant software "judith" handling the http posts and notifies me if soil getting dry or outdoor temperature drops below lettuce's acceptable range. an 12v/7ah ups battery powering it about 8 days with 30 minutes of send interval. planning a solar charger with li-ion battery for it in my next spare time.
 
 Cost table
 + ESP-01 wifi board: $5
 + Arduino Uno R3 (clone): $3
 + HL-69 soil moisture sensor: $2
 + ds1621 thermometer module: $2
 + light resistive diode: $0.5
 
 took 3 days to develop, 3 weeks to fix errors. for sake of time don't buy chinese clones of esp8266. an extra $5 can save your whole month. if interested, source code available in devzone section and github



Older Entries