Wednesday, October 19, 2016

Headers, Caching, and Cookies: Oh my!

I had to pound my fists on the table again when the topic of cookies, caching, headers, and all sorts of jargon popped up when talking about the tech of webapps.  Webapps, like online shopping cart apps, are one of those concepts that are simple on paper but can easily become convoluted, especially when terms get swapped around or poorly defined.  So I'm running down the list based on experience.

Since this is pragmatist approach, not theoretical smoke & mirrors, I'll freely refer to common software implementations of the concepts, including Apache HTTPD, JBoss/Wildfly, and the all powerful cURL.

Back to Terms & Definitions

Back to basics.  Web services are almost entirely based on a protocol that was purposely designed to NOT maintain the state of any communication.  aka. HTTP.  "It is a generic, stateless, protocol"[1].   Technologists have, since the original inception, extended and modified HTTP to enabled stateful mechanisms. :sigh:  Those modifications include:

  • Stickiness - "stickiness" is nothing more than than an antonym for "load balancing".  In other words, you actually _want_ your web client to be serviced by a particular server (affinity).  Implementations of stickiness are often done with HTTP Headers!  :gasp:  Not especially what they were intended for (see Web definition) but implementations in Header include:
    • Cookie - statefulness; https://tools.ietf.org/html/rfc6265
    • Sessions / SESSIONID (JSESSIONID/PHPSESSIONID) - more statefulness
    • jvmRoute - mod_jk / mod_proxy[2]  
  • Authenticated Sessions - obviously not the same as a bare bones web session; anyone who tells you that all you need is to put everything, including sessions, behind HTTPS is full of security blackmail.  TLS/SSL over HTTP has nothing to do with authenticating the session; it may (though not always) be authenticating the communication from client to server -- but that is it!  An Authenticated Session is not achieved by HTTPS.
  • Caches - not to be confused with Cookies; client-side (web browser) cache[3] -- often mislabelled as "cookies"; server-side (web site) cache -- mod_cache
    • Cache everything!  Actual web services payload may vary.  Based on data from one of the largest content delivery services for the Web, Akamai, the top 3 culprits slowing down web payloads are: 1) images, 2) images, and 3) images.
  • Load balancing - only makes sense when talking about volume (throughput); has no bounded direct link to latency!  You can have a well balanced load that includes a session that is simply slllllooooww (latent).  That in no way means the lb (mod_proxy, mod_cluster for Apache HTTPD) is broken but could very well mean a stuck thread on the backend JBoss or DB.










References

[1] https://tools.ietf.org/html/rfc2616. versus  https://tools.ietf.org/html/rfc6265
[2] http://tomcat.apache.org/connectors-doc/reference/workers.html
[3] pg. 108 / Section 14.9 Cache-Control, https://tools.ietf.org/html/rfc2616#section-14.9


Security: Smartphones v PC: Deja vu

My officemate's smartphone was stolen.  By the time she got online via her laptop (because she was nowhere near home), the remote lock or wipe couldn't find her phone.  It had been stolen/wiped.  

Smartphone security paranoia popped up again in conversation when a friend overlooked my shoulder and saw how ridiculously long it took me to type my password.  He quipped:

"You know that's not secure".
I said, "I've encrypted the phone."
"Oh I can get around that", he said.
"You mean the USB connection and an Android debugger?  Yes I suppose you could eventually brute force it but that will take awhile.  By then, I'll remotely wipe the phone."

My hope was to avoid a debate about fundamental problems with IT security, and share my belief that a simple risk/reward exercise (including the annoyance of securing a tech device versus its usability), should result in enough security to give hackers at least a big headache if not full on despair.  My belief about consumer electronics security is: hackers with a specific vendetta against YOU personally -- so someone wanting to ruin your life -- will invest considerable effort in tearing down layers of security; versus hackers at large looking to exploit maximum reward with minimum effort by targetting smartphones/PCs/etc en masse will skip over consumers with decent layers of security.  In other words, most folks have more to fear from their closest friends and family -- who already have access to private or personal information about you anyways -- than anonymous hackers who only know you as an IP address, but only IF you've dotted your i's and crossed your t's.

So smartphone security should target two camps:
  1. local / physical access
  2. remote access


Assume an ex stalks you online, goes covert to get a job as a repairman so she gets access to the building you work at, secretly stalks you at work to figure out when you occasionally leave your smartphone on your desk --, and nabs your phone while you're in the bathroom.  OK now your ex has local / physical access to your smartphone.  Only a few common security "states" exist for any smartphone:
  • unlocked screen
  • locked screen
  • connected phone
  • disconnected phone


The interesting thing to notice about this list smartphone states is they apply to computers generally, especially the old PC from yesteryear.  Desktops, laptops, smartphones, etc. consumer technologies have many security traits in common, and this commonality means that basic security concerns for computer technology in general applies to smartphones specifically.  A PC from 20 years ago, this list for solutions were:

screensaver w/ password
disconnected PC (no LAN)
encrypted disk

20 years ago, it was silly to leave a computer screen unlocked, and trivial to get data from the computer if its disks were not encrypted and someone, like your ex, still had physical access to your PC.  The same applies to smartphones today.  

My friend's theory was a hacker just inserts a USB cable to bypass a smartphone password.  Google Android smartphones have had local storage encryption since 2011, and Apple iPhones have had local storage encryption since 2009, so both major smartphone manufacturers finally caught up to Blackberry RIM (which had been encrypting smartphones since insert date to close that security hole.  (One technical point: Android still does not encrypt external or expandable storage --  SDCards or any storage media that is not primary storage -- but Blackberry's later OSes did encrypt removable storage media.  Another point of digression: Blackberry smartphones so were highly regarded for their security than some governments outlawed their sale or used their own intelligence agencies against Blackberry's secure communications, such as India.)

Seeing the same risk profiles repeat through history, and techies giving newer yet similar security solutions, yet consumers facing the same pitfalls, I ask myself: have we learned anything as consumers of technology??

Security: a Techie's Steps for Safely Browsing Nowadays



I use these steps for safely browsing nowadays.  Ordered sequence matters!

  1. Verify eMail Anti-spam/filter - eMail provider (Gmail/Outlook/Yahoo), eMail client (Outlook), 3rd party (McAffee/Norton/Avast)
  2. Enable automatic updates - OS (Windows, OSX, Android, iOS), Apps (Windows Store, Google Play/Store/Apple iTunes/Store) 
  3. Enable secure web browsing - web browser (Chrome/IE/Firefox) - TLS, no SSL3 
  4. Use Multiple, difficult passwords - 3tier pyramid approach
  5. Use complex online account hints - the answer to "your first pet's name?" isn't actually my 1st pet
  6. Enable 2FA (Two Factor Authentication) Online 
  7. Secure 2FA devices - encrypt smartphone/tablet, remote erase lost smartphone/tablet - iOS/Android
  8. Encrypt local storage - laptops, tablets, smartphones
  9. Encrypt cloud storage - Google Drive, Microsoft OneDrive, Apple iCloud
  10. Filter/Block online ads - web browser extension - ABP/uBlock 
  11. Verify public profiles - Pipl
  12. Setup local anti-virus - Avast/McAffee/Norton
  13. Setup local anti-malware - Malwarebytes

An Open Complaint Against Web Analytics and Advertising

An Open Complaint Against Web Analytics and Advertising


I’ve grown increasingly dissatisfied with web browsing speeds but not for the usual reasons.  Usually folks complain about staring at a screen that is barely able to load a website because they’re at a congested cafe whose public Wifi is overwhelmed or out at a remote beach where their signal is down to the dreaded “1 bar”. That’s not my complaint.  Instead, I blame a deluge of web analytics and online advertising for reducing my browsing experience to digital snail speeds.  


The Modern Website


When I browse to any modern website — well OK the choice pick would be a retailer’s website that’s full of ads — I look around the edges of my web browser as the page loads.  For example, my web browser is Chrome so when I browse to target.com the bottom of Chrome shows “waiting for facebook.com” and other websites besides Target’s.  Facebook and Twitter are just two examples I notice but it seems like a deluge of websites stream across the bottom of my browser while Target’s homepage slowwwwwwly appears on my screen.  I know what technology is behind these other sites loading but I wanted to get the scope of its impact on browsing, so I downloaded a web browser plugin called “Ghostery” to tell me how many other websites were hit when I went to target.com.  My browser hit over TWO dozen websites just to browse to the target.com homepage.


Modern Web Technologies


For full disclosure, my browser wasn’t loading over two dozen websites in their entirety but don’t let advocates of this technology trivialize it’s behavior as “simple hops” made while going to a website.  These “simple hops” are reducing my browsing experience to snail speeds — or as we say in the business: “causing latency”.  This latency behavior is the affect of two modern web technologies: 1) web analytics, and 2) online advertising. These technologies enable digital marketing, including online campaigns, and targeted advertising.  Both of these web technologies have been in the spotlight by privacy fanatics because of their intrusive personalization of websites but that’s not my complaint here.  I’m a huge fan of capitalism and marketing because I’d rather see an ad about a Star Trek movie than a new bra.  My complaint is the negative experience I’m having because of the sluggishness these technologies are causing when I’m shopping online.  In other words, my online shopping experience is taking a “hit”.  The irony with a negative online experience from analytics and advertising is that both of these technologies are intended to enrich our online shopping experience.


Latency Check


In their defense, advocate of web analytics and online advertising will say: a) the “hops” should have been optimized to reduce latency and b) the value of marketing results and targeting advertising outweigh any latency.  The later (b) defense is a slap in the face for criticizing technological advances so I’ll simply ignore it.  The former (a) defense is technologically sound advice but, sadly, it isn’t working.  Using another website tool, Pingdom, I made a cursory check of the time these hops burn up with hitting Target’s homepage.  It took 2.5 seconds to get target.com (http://tools.pingdom.com/fpt/#!/jFbn7/target.com).  The latency Pingdom calculates excludes the additional time your web browser needs to display that homepage, or what technologists call “rendering time” (when your browser renders the page to your laptop or phone), so any time seen by checks like Pingdom imply an even longer period of time before a person browsing can interact fully with the webpage.  


Again, defenders of analytics and advertising technology will chastise me for ignoring the Elephant in the Room: the biggest latency comes from getting and rendering media, like images, because big pictures must be downloaded to the browser so you can see them.  They are right but the Devil is in the details.  For example, the biggest chunk of time spent getting target.com comes from images (65% of time).  Web analytics and online advertising are embedded in the webpage as scripts and like images must be downloaded but executed by the browser instead of displayed.  In this example, Pingdom also found that the total size of images is not much bigger than the size of web analytics and online advertising (842kB versus 698kB*).  So target.com has nearly as much scripting payload as it does images.  There could be scripts other than analytics and advertising but another indication about the cause of latency comes from the amount of time Pingdom’s browsing spent “connecting” and “waiting” rather than receiving the Target’s homepage.  Less than half of the time I’m waiting to click around for a new coffee maker on target.com is when my web browser actually receives the webpage.  Speaking of “Devil”, as a comparison I pointed Pingdom against the old Internet Explorer is Evil webpage (http://toastytech.com/evil/) … that webpage only took 306 milliseconds to get  (http://tools.pingdom.com/fpt/#!/bNJGe7/http://toastytech.com/evil/).  Ghostery also found zero web analytics or online advertising on the Evil webpage.


A Magic Pill?


A smart technologist or intuitive advocate of these technologies would expect that a web server should be able to respond faster by working on components of the webpage in parallel.  I would agree.  When viewed with a very broad lens, modern website design could be described as distributed computing because webpages distribute the analytics and advertisements to agencies that specialize in these technologies.  Web developers simply embed these agencies technologies into their website as scripts instead of maintaining all these technologies from their own web server.  Yet a broad lens overlooks the details and complications.  After digging around in the world of webpage design and digital marketing, it appears parallelizing webpage rendering is difficult.  Evidently modern browsers render webpages the way a fax machine scans — so in sequence from top to bottom — so the latency problem first hits when something in this sequence takes a long period of time.  A long running step, like a web analytics script embedded in the webpage, will block the rest of the webpage from rendering until the agency’s web server responds and the web browser executes their script.  You’ve experienced the results of this “scanning” latency when half a webpage appears on in your browser, when bars or menus appear after a center panel is already visible, and when images appear later than text.  Web developers have some tricks up the sleeves for circumventing this blocking behavior, such as event handlers and optimizing the webpage sequence.  For example, scripts that execute advertisements and analytics could be embedded at the bottom of webpages and not block other components from rendering so I can start interacting with the website.  (I’m no expert so one good source on scripting behavior is at: http://mrcoles.com/blog/how-tracking-scripts-affect-page-loads/)  But this isn’t a magic pill.  Web browsers react differently to event handlers and even then the script response time is only as fast as the agency’s web server.  So another technology, content delivery or edge networks, is introduced to further optimize the typical response time of web servers.  Content or edge networks work well but the latency persists because of scale.  


1 Bar


Websites are embedding a deluge of analytics and advertising scripts in a single webpage that the gains in event handlers and delivery networks has become moot.  Again, target.com called over two dozen external web servers for a single hit to their homepage.  The strongest indication of a degrading problem is modern hardware running modern software rendering a modern webpage slower than a decade old webpage.  My simplistic comparison of time and payload above hints at this being the root cause.  Although I see great value in tracking campaign efficacy, adjusting to customer browsing behavior, and other datasets that web technologies have enabled with web analytics and online advertising, I don’t see the value of these technologies outweighing a sluggish online experience.  Our online experience is being smothered by marketing technology.  I might as well be browsing with only “1 bar” of signal.

... to be continued.

Predecessors of PaaS - DRAFTv1



Predecessors of PaaS - DRAFTv1


In a datacenter far far away, there was "pre-PaaS"...


The pessimists of Cloud and stalwarts of IT should be happy … for now.  Gartner’s hype-o-meter put Cloud concepts in decline (or rather the “Trough of Disillusionment”.  Optimistics will note that  -- at least according to Gartner -- after a period of disillusionment, technology re-emerges in a stage of enlightenment, followed by stable adoption.  So let’s do the timewarp again.


Way back in 2010, some smart architects on my team finished implementing an application management and orchestration platform that would rapidly change user expectations.  That platform's sales slidedeck, dated 2008 [1], had neither the term “DevOps” nor “Paas” (or any of the *aaS Cloud terms) but “Cloud” is mentioned, although fewer than a dozen times.  The product’s slidedeck wasn't filled with modern buzzwords but focused on 1) dynamically orchestrating virtual resources and 2) automating application deployments.  


[1]


I joined that smart team of folks when our goal was to streamline and automate processes that made fast, incremental releases of a distributed, multi-tier trading desk system finally possible. Many of our peers from traditional teams, like Dev and Ops, didn't' really get why we sat inbetween them. It was simple: because the traders demanded changes to their financial products ASAP.  The traditional technical teams could not deliver feature requests at a pace that satisfied the business needs, aka. the traders in our case.  Users of the system would request a feature or fix through typical business analyst channels and WEEKS or even months for the development lifecycle to finally spit out their change to production.  By 2010, our application deployments were mostly automated, and though the final delivery to Prod wasn't yet push button, we really had streamlined the process. We had DAILY AUTOMATED builds.

I had heard the term “DevOps” for the first time while working there and really didn't know what it meant, and our architects faced many cultural obstacles because we were the new kids in the cubes.  Many of those cultural obstacles boil down to ongoing silos in organizations but the technology itself has now graduated to getting its own label: PaaS.


Ideas/References:


The reason rapid deployment became a necessity is because small to medium sized businesses want to reach the biggest market / "market capture".  These 21st century, technology dependent customer’s no longer accept that any application’s functionality would be interrupted by a large maintenance window set during U.S. “off hours”, nor that productions and solutions would not geo-specific.


Skeptics of PaaS: early on in 2008 Reese blasted the de facto feature of auto-scaling that’s in most IaaS and PaaS solutions. http://broadcast.oreilly.com/2008/12/why-i-dont-like-cloud-auto-scaling.html

Velocity 2009: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr


Technical and sociological justification of continuous delivery with PaaS: McCabe's Cyclomatic complexity, and Conway's Law. Cyclomatic complexity - modular programming, use small components together; Conway’s Law:

“If the parts of an organization (e.g. teams, departments, or subdivisions) do not closely reflect the essential parts of the product, or if the relationship between organizations do not reflect the relationships between product parts, then the project will be in trouble... Therefore: Make sure the organization is compatible with the product architecture"

Being Online: Wolf in Sheep's Computer - DRAFT

Being Online: Wolf in Sheep's Computer - DRAFT


Even technologists such as myself are not immune to hackers.  Two of my credit cards were replaced because of the back-to-back Target and Home Depot hacks.  I’ve avoided outright fraudulent purchases and identity theft but I know folks who've succomb to these. The essential rule for safely browsing online is: perseverance. With new technology come new tricks that protect your modern, online browsing. Gone are the days of simply installing anti-virus and changing your passwords. I’m sharing some these latest tricks here in the simplistic way I can. Let's start with a "Do / Don't Do" list, with more justification as an Appendix.


First up -- Passwords are OUT! DON'T just change your passwords.
  1. DO setup Multi-Factor or 2-way Authentication, DON'T just use passphrases
  2. DO keep Passphrases in Your Head, DON'T blindly trust password managers or vaults
  3. DO Pay Online with Layered Accounts, DON'T pay via Debit Cards, Checks, or Bank Accounts
  4. DO pour on Layers of Security, DON'T just trust one website
1. Multi-Factor or 2-way Authentication


Over a decade ago, technologists productized something that supplanted the old school username/password paradigm.  Technologists have blasted passwords as a single point of failure for a long time. It's a weak form of protecting your identity online. We introduced 2-way or Multi-Factor Authentication (2FA/MFA) because it added a 2nd (or Multiple) way for you to login.  Imagine 2FA/MFA with this analogy:


You walk up to a closed door with key in hand.  You try to turn the door handle but it’s locked so you put your key in and turn.  When you turn the handle and push the door, the door doesn’t open.  Instead you hear a latch open up next to your ear and see eyes peering out at you through a slit in the door that the latch opened.  You hear the girl behind the door say “Violet”, so you say “Yes”.  Now you hear the girl turn a hidden deadbolt on her side of the door.  You push the door again and it opens!  


Putting your key in the door and turning its handle is the old school username/password paradigm.  The girl saying a secret word that you acknowledge and her unlocking a deadbolt on the hidden side of the door is the new paradigm: 2FA/MFA.  You cannot get the door open with just your key but must also let the girl behind the door see you and respond correctly.  Pretty cool!


Modern technology has enabled common devices, like your phone itself, to be a 2nd way of authenticating online because you almost always carry your phone around.  Your dumbphone or smartphone receives a secret code, either via SMS txt or a mobile app, when you attempt to identify yourself online.  You use this secret code from your phone alongside your username/password in a typical website logon.  This new sequence means that a hacker must both: 1) find your username/password, AND 2) steal/unlock your phone.  

It's also important to note that the secret codes we’re discussing are temporary, unlike passwords that are seldom changed. They're much like the 007 motto "this message will self-destruct" because a hacker doesn’t have the right secret code if they look over your shoulder or make a guess like they can do for passwords.  These 2FA/MFA secret codes are quickly randomized, usually every minute.

A Picture is Worth every Pixel


A phone isn’t always required.  Some 2FA/MFA technologies use your web browser as an alternative to make the 2nd verification without sending a secret code.  These browser options include asking you to verify a picture code or to answer personal questions, like “What is the name of your preferred charity?”  Word of caution on setting up these alternatives to secret codes: personally identifiable information (PII) is not a good choice for 2FA/MFA setup.  A hacker can usually figure out your PII.  For example, figuring out where you were born is trivial so that question should be avoided.


2FA/MFA provides another layer of headache for a hacker to ruin your life. Gone are the days of simply securing your online identity with a “strong” password.  Sadly, online technologies have only recently begun to adopt 2FA/MFA after mulling around the elite halls of computer nerds but I’ve found the most popular online services have gotten aboard.


Here are popular online websites that allow you to setup 2-way or Multi-Factor Authentication via your smartphone:


Social media:
Facebook
Twitter
Apple ID
Google+
Linkedin
Microsoft Live


Banks:
Bank of America
JP Morgan Chase
Barclays


Payment processors:
Visa
Discover
Paypal


Cloud storage:
Dropbox
Evernote
iCloud
Google Drive
OneDrive/SkyDrive


(*from personal experience and from https://twofactorauth.org/)


2. Passphrases in Your Head


"The 4 frogs farted!" is a silly phrase that is more secure than "Jr1981" as a passphrase. Even spaces are valid in passphrases, hence technologists prefer call your login credential a "passphrase" instead of "password".

A Pyramid of Passphrases


Keep passphrases in your head. I'm not abandoning old school password tips entirely but don't let 20th century best practices give you any kind of comfort. We are dealing with a whole new set of technology in the 21st century. Any password that is written down, even in your Evernote/Dropbox/Drive digital notepad, is a sitting duck for hackers.  

You should be grumbling about having too many passwords to remember, all while IT nerds demand that you keep creating more and more of them and higher and higher complexity!  I’m one technologist who admittedly recommends using only as many passphrases as you can actually remember. Why? There is no reliable way to forcibly extract a passphrase that is just a bunch of neurons in your brain. There are means, even if difficult or improbable, to break into locked drawers full of password notes and even hack into password managers that magically hold all those passphrases for you. Just keep passphrase management to yourself.

I recommend creating a 3 tiered pyramid of passphrases for all your online activity.  

Bottom Tier

Imagine the bottom of this pyramid being a ton of websites that you visit that require you to create some kind of account but keep very little or even no personal information about you. Online forums are a great example of this bottom tier of our pyramid. Many online forums don't let you search their topics or threads unless you create an account on their website, even if that account doesn't require your address, or age, or really much of anything other than a username. Keep a throw away passphrase for this bottom tier of websites.  

Middle Tier

Next is the middle tier of the pyramid. Here sit a sizable amount of websites or apps that require some personal information about you. Most social media sits here, both in website and app forms. Create a

Top Tier

Finally, the small top tier of your pyramid. These are your online crown jewels -- Bank websites and apps, for example. These websites and apps not only require and maintain personal information about you but access to them has real life affect, like paying bills, filing claims, etc. Some may put their social media accounts in this tier instead of the middle tier. I'm making no rules but the model should fit your risk tolerance. The less tolerance your life has for someone hacking into a tier -- say a violent ex hacking into your Facebook -- the more reason to put that website or app into your top tier. There is one caveat: you must trust your top tier with your most confidential information. If you don't trust the institution or their own online presence, then your account with them should not be part of your top tier. Take online credit card accounts as an example. Credit cards are famous for not only informing their consumers of fraudulent activity but many have even limited your peronsal liability for fraudulent charges with legalese. I've found their commitment to securing your account with them in my credit card contract. For me, that kind of legal commitment implies trust in their online account security.


4. Layers of Security

Try hacking into yourself online. Ask a flesh-and-blood friend of yours to "Unfriend" you from Facebook, Google+, Twitter, or some other social media. After being "unfriended", see how much information your flesh-and-blood friend can find out about you via their own account. This exercise is simulating how online hackers find personal or private information about you, and the results from this self-hack will shock you. Yet, there's plenty of fixes available that don't require the Nuclear Option -- you don't have to go offline. Go back into your social media account and fix the privacy or security settings around each bit of personal or private information your "Unfriend" found. After fixing up your privacy and security leaks, try hacking into yourself again. Rinse and repeat until you've clamped down on any information leaks.


MasterCard SecureCode

Verified by VISA
http://usa.visa.com/personal/security/security-program/verified-by-visa.jsp?n=1

Masquerade Cards


Bank of America has an online service that generates an ad hoc, temporary credit card that is backed by your actual credit card.  This service, called “ShopSafe”, ensures that the online merchant’s payment system never actually sees your real credit card data.  The Bank processes the transaction on their side by acting as a proxy for your card.  This service can also generate a “masquerade” credit card data for recurring, online bill payments.


Pictures at Logon


Yet Another Layer


Services using Paypal Payment /  VISA Checkout


eBay
HuluPlus
Netflix

Paypal also offers a Credit Card to front any payments that accept credit, which is basically everyone.

Tuesday, January 5, 2016

Internet Speed

Internet Speed tests without Adobe's bug laden Flash and even with tests for measuring traffic shaping:

http://www.measurementlab.net/tools/ndt

http://beta.speedtest.net/

Wednesday, December 30, 2015

Editing JSON (Cloudformation)

Cloudformation scripting is terrible because it's based entirely on editing JSON.  Ugh.  Human UNfriendly.

I'd tried several editors.  Amazon's editor (for Cloudformation), online editors, and plain ol' vi.

Props to this guy for extending JSON editing for my friendly goto editor, vim:

https://github.com/elzr/vim-json

Wednesday, August 12, 2015

SSH, Yes!

SSH tunneling to the rescue!  Yes!

Circumventing some local firewalls with port forwarding :)


ssh -N user@box-i-can-access localhost:5432:box-i-cant-access:5432

Thursday, July 16, 2015

IE9 versus PCI compliance

Nerd speak: Starting July 2016, HTTPS communication of confidential information must be negotiated as TLSv1.1+ for PCI DSSv3.1 compliance.

Layman speak: Your Grandma's computer won't be able to buy stuff from Etsy.  (This effectively means the web browser that first shipped in 2011 with Windows Vista, IE9, will no longer be supported for credit card payments.)

Sincerely,

Your Paranoid Computer Nerds

Migrating_from_SSL_Early_TLS_Information Supplement

Thursday, July 9, 2015

HTTPS: What Nobody Told Us

I burned one too many hours troubleshooting an HTTPS issue and decided to share lessons learned.  Both programmatic no-no's and TLS details that nobody told us.

The Audit

This story started when some auditor got in a frenzy that TLSv1.0 was allowed for public HTTPS communication with a customer's web application.  That auditor demanded that only TLSv1.1 or v1.2 be allowed despite version 1.0's problem being isolated to weak CBC and RC4 ciphers (aka. the infamous BEAST attack) -- ciphers that we already weren't allowing.

Side rant: who has time for audits with little security justification?  I've no patience for security "gurus" who cannot run a simple scan to verify their worst nightmare or best dream:

justin:tmp jpittman$ nmap --script ssl-enum-ciphers $HOSTNAME

Starting Nmap 6.40-2 ( http://nmap.org ) at 2015-07-08 11:12 CDT
...
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https
| ssl-enum-ciphers: 
|   SSLv3: No supported ciphers found
|   TLSv1.0: 
|     ciphers: 
...

OK fine, assuming the auditor's request is legit, I disable TLSv1.0 at load balancers terminating HTTPS ... and the webapp breaks.  

For better disclosure, I should clarify secure communication in this particular design.  This was a typical, 3-tier application architecture that was designed so that a layer of load balancers would proxy requests between the set of front-end tiers, aka. on behalf of web servers and app servers, yet there was also intra-network communication between application servers that hosted different apps that these load balancers also proxied.  This design simplified HTTPS termination because only the load balancers acting as proxies needed to have their secure certificates managed, however it did complicate the idea of one app "server" making a client call to another app server within the same network.  Also, these deployed webapps were all Java based -- but programming language really only matters for details in implementation.

The Error

When I turned off TLSv1.0 on all the load balancers, one of the deployed Java apps acting as a web client started throwing errors about SSL handshaking, like this:

IOException when getting the response content input stream javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

Here's the first part of the story that nobody told us: misnomers.  This error makes it sound like an SSL protocol is failing when, as I'll show later, it is a TLS protocol failure.  And this misnomer isn't particular to Java.  I checked Python, Ruby, and PHP only to find that they too negotiate TLS protocols via inaccurately named "SSL" libraries or methods.  Sure, you can make a defense for using the term "SSL/TLS" in documentation but what a way to conjure up a red herring.  I wasted cycles thinking SSL protocols were still in play when they never were!  Shame on you! 

A Secure Socket  

By digging into Java code samples, I found some ideal tests for uncovering the root cause of this ill-named handshake error:

SSLContext context = SSLContext.getInstance("TLSv1.2");

Now I'm no Java programmer but this bit of code is fairly simple.  It says the SSL/TLS context, so in this case a client call, will be gotten with a parameter that sets the TLSv1.2 protocol.  That client context creates a socket to some server -- hence the acronym Secure Socket Layer (SSL) of the original, now inaptly named class SSLContext.  When I tested this code with the parameter TLSv1.2 the proxied connection to the problematic server worked, but when set to TLSv1.0 the connection failed with the above SSL handshake error.  Ahah!

TL;DR


Some good ol' TL;DR documentation verified default Java behavior that would explain the errors too.  This case used Oracle's Hotspot JVM and luckily that vendor's documentation is usually verbose, if not also cryptic.  I read Oracle's rather lengthy reference guide to Java Secure Socket Extension (JSSE) that covered both those SSLContext and HttpsURLConnection classes.  First off, picking the correct version of the documentation avoided some false fixes.  Java 8 fixes didn't apply for a case of Java 7, as this was.  Next, the mode of the JVM as client versus server altered its default behavior.  Oracle said JVM clients enable a different set of protocols and versions than those in server mode.  Also, the documented samples set SSLContext to "TLS" inline -- which I would assume could mean any version of TLS -- yet the documentation clearly says that "TLS" means "TLSv1.0", excluding v1.1 or v1.2.  If a lazy programmer didn't read that documentation yet borrowed its code samples, then she would have actually hardcoded the client to TLSv1.0 versions.  Finally, the SSL handshake error would occur if the web client used an SSLContext method that set buildtime configurations of the protocol.  

So hardcoded bug or default behavior? 

A Web Client

A similar bit of Java code shed more light on the common world of this web client problem:


url = new URL(https_url);

HttpsURLConnection con = (HttpsURLConnection)url.openConnection();

In this code, an HTTPS connection is created to a URL of some server, aka. a classic web client.  Yet this HttpsURLConnection method doesn't specify the protocol or version like the previous spinnet of code.  I had read that HttpsURLConnection honors the JVM runtime option https.protocols to change protocols and versions.  Here is where runtime versus buildtime components revealed themselves as part of the problem.  By testing the means to setup a secure web client via two different methods -- socket versus URL -- it became dramatically clear that I was probably dealing with a classic, hardcoded bug.  

I reconfigured the runtime of this HTTPS connection to use TLSv1.1 or v1.2 and my Java test client worked!

Hardcoded / Buildtime Configuration  

A developer confirmed the root cause was indeed a hardcoded setting of the protocol and version when he changed the problematic webapp client's SSLContext parameter to "TLSv1.2", rebuilt, redeployed.  Although the fix was simple, I had burned too many hours troubleshooting -- what was essentially -- a hardcoded / buildtime problem that could not be trivially fixed with runtime changes.  I had followed some red herrings while searching for the root cause of this seemingly simple SSL/TLS change, including:

  • Server/Client Certificates
  • Certificate Authority chains
  • Firewalls

So nerds beware.  You may need to go down the rabbit hole far beyond the typical diagrams with labels like "SSL Handshake".

References

https://tersesystems.com/2014/01/13/fixing-the-most-dangerous-code-in-the-world/
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https
http://www.oracle.com/technetwork/java/javase/documentation/cve-2014-3566-2342133.html
http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/keytool.html


Tuesday, June 2, 2015

Security: Unexpected Terminal Event

Today I killed a production website by pointing a webapp vulnerability scanner at it.  The unexpected stress brought the appserver to its knees while revealing some holes.  Luckily my more tactical colleagues came to the rescue and had the website back up in minutes -- after I had killed the scan -- but this opened two cans of worms. 

An obvious need for the design and implementation of 1) stress/load testing, and 2) vulnerability/penetration testing in regular operations.  These (2) needs are usually ad hoc. Both need to be part of processes in certification for release.

On another note, I was using the awesome open source tool called w3af.  :)

Security: HTTPS Utilities


I was going to talk about recent hacks against "secure" web communication, aka. HTTPS (HeartBleed, POODLE, Beast, etc.) but that is a bloated topic.  Instead, I'm just going to demo 3 invaluable utilities for techies specific to TLS/SSL, and show how Amazon makes managing HTTPS so simple that technologists have even more reason to be lazy.

3 HTTPS utilities

  1. openssl
  2. curl
  3. nmap
That's really it.  These utilities have been around for years so there's nothing new here.  A few examples will demonstrate their utility in verification, identification, and negotiation of HTTPS communication.

Identify all secure communication options serviced by website:

$ nmap --script ssl-enum-ciphers www.httpvshttps.com
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https
| ssl-enum-ciphers: 
|   SSLv3: No supported ciphers found
|   TLSv1.0: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
...

Verify secure communication options negotiable by client, specifically the highest TLS version with RSA authentication and keys, and high AES encryption:

$ openssl ciphers -v | grep 'TLSv1.2' | grep 'Kx=RSA' | grep 'Au=RSA' | grep 'Enc=AES(256)'

AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256

Negotiate a pre-defined secure communication to simulate a web client:

$ curl -v --location --tlsv1 --ciphers AES256-SHA https://www.httpvshttps.com

Amazon HTTPS & ELBs


AWS ELBs (Elastic Load Balancers) make managing HTTPS so simple. These load balancers can be setup with user-defined certificates to terminate secure communication to a website, and are deployed with either pre-defined or user-defined security policies.  When a customer asked me to disable TLS v1.0, I simply change their ELB security policy by removing all the ciphers available for that kind of secure negotiation via a checkbox:



Of course, no good technologist trusts a GUI so I verified that change by using the 3 utilities above :)



Wednesday, May 27, 2015

Security: Hactivism

Imperva released a whitepaper detailing an "activist" hack that mimics many of the means I'd outlined in dissecting the Sony attack last year.  Not coincidence.

Anatomy of a Hacktivist Attack

Wednesday, May 13, 2015

Internet: Authoritarian Machine

The "Internet" is, fundamentally, authoritarian but appears unfiltered and inclusive on the surface.  I don't say that with any malice.  Your Internet service provider, for example, wants your money so long as you aren't a hot potatoe.

What I mean by authoritarian is how the "Internet", a rather loaded term, can be heavy handed in squelching participation that inhibits its normal operation or societal rules.  A remarkable example of this is the infamous Dark Net, and a more poignant example is China (which I'll come back to). Hosting providers are "blind" enablers of a myriad of online activities, including those considered amoral ... Avenue Q's character Trekkie Monster humors us into realizing this ..., but sometimes ISPs can't be bothered to enforce societal rules until a participant gets themselves caught or disturbs normal operations.  Once they've got a hot potato on their hands, any Internet provider is going to pass on the buck.  This explains all the liability crap that we sign off on when buying an Internet service.  I don't mean to imply you or I or anyone is doing anything illegal with the Dark Net example, but I'm saying something as benign as free speech is actually a societal characteristic of the US.  It's up to participants of the Internet to follow societal rules lest an authoritarian machine works against us.

The Internet has not created, despite popular belief, any (direct) relief to societal problems and is, at least technologically speaking, highly conformist.  Those technologies that form the Internet, so to say, are a myriad of protocols and standards created (or "authorized for use") by various quasi-government committees (IETF, IANA, ICANN, W3C, ISO, etc.) and those committee members seek conformity, whether for good or bad.  Even technologists use telling terms that allude to the Internet's style of governance, like "Certificate Authority" and "authoritative name server" (despite preying on people's idealizations with mumbo jumbo like "Web of trust").  That conformist characteristic is a fundamental misunderstanding overlooked by Internet free speech and civil rights advocates.  Any variance from standardized or expected means of communication and connectivity is both technologically prohibited and authoritatively regulated via the Internet.  As long as Internet participants do not undermine the operation of those technologies (i.e. writing a deluge of spam or flooding a website with noise) then you're OK, otherwise folks are labelled a hot potato, aka. hacker, get their connection blocked, and maybe their "private" information sent off to law enforcement.

Mainland China stands out as a poignant example of what I mean.  It's not like the "Chinese" Internet is different than our own.  It's the same technologies (HTML, HTTP, TCP, IP, OSPF, etc.) applied in an overtly authoritarian way to enforce a different set of societal rules than in the US.  If we were Chinese, those Internet technologies would enable you to create an inflammatory website with Mandarin characters, encoded in a standard format (UTF-8), but your local Communist mayor would shut down your site and block your access to facebook.com.  That's a characteristic that is in opposition to the folksy (mis)perception, especially in Western countries, of the Internet being inclusive and unfiltered.  And, of course, there's always the NSA in the U.S.A.

In some ways the Internet reminds me of the Matrix: when there's a bad apple, like Neo was, it just throws the apple away.  That ability to squelch free speech and infringe civil rights (if those exist in a society) are merely authoritarian actions enabled by the Machine that is the "Internet".  Thankfully, the above mentioned quasi-government committees that affect Internet technologies have usually taken a hands-off approach to societal issues, so we have a ton of free speech happening on top of Internet service provided in the US.  Town hall debates still happen but now they are flame wars in forums -- hopefully encoded in UTF-8 but probably in ISO-8859-1 since we're in the West and largely forget the rest of the world :)


Tuesday, May 12, 2015

Security: Hacking Wordpress/PHP, Ruby Comparisons, and Lessons Learned

Another US Cert alert went out for Wordpress last week.  I'm not shocked.  I call Wordpress one of the biggest hacker honeypots around because it keeps popping up in cybersecurity news.  Yet, it's not fair to be subjective.   To be more objective, I'll take a look at verified  software vulnerabilities to assess what's happening under the hood.

Part 1: Lots of Vulnerabilities


I went back to MITRE's CVE database again (NIST's mirror isn't as user friendly) to compare how many vulnerabilities have been occurring in Wordpress and PHP -- the core language behind Wordpress -- versus another popular web development language, Ruby (and Ruby on Rails).  See my previous post on CVEs and preventing software vulnerabilities for using these databases.  Anyways, a simplistic comparison of all vulnerabilities found in Ruby versus PHP is staggering:

CVE Total Counts

$ curl -s https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ruby | grep "CVE" | wc -l
288
$ curl -s https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=php | grep "CVE" | wc -l
5812

For the studious, since Ruby doesn't really have a good equivalent to Wordpress, it's fairer to see that Ruby on Rails CVE counts are not on the same scale as PHP.  Here's their CVE dumps of those too:

$ curl -s https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=rails | grep CVE | wc -l
122
$ curl -s https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=wordpress | grep CVE | wc -l
951

It should be noted -- especially for the skeptical -- that PHP and Ruby have some relevant differences.  PHP version 1.0 was released in June 1995 while Ruby 1.0 was released December 1996, but PHP is only slightly older than Ruby.  The CVE dumps above are for all known time, so I don't think their age explains the difference in quantity (and the CVE databases only go back to the early 2000s anyways).  I chose to compare these two languages instead of solutions built on top of them, like Wordpress, because they are both popular, both interpreted languages, and both end on the front line of cybersecurity -- running public websites.

Adoption and Presence


A coworker sent me over to BuiltWith for an unrelated comparison of website softwares so I thought it would be a good tool to dig deeper on the software adoption being related to quantities of vulnerabilities.  BuildWith is a service that scrapes online footprints to determine which softwares run public websites -- everything from operating systems to content management systems like Wordpress.  It's pretty nifty.  I asked BuildWith what the adoption rates look like for PHP and Ruby on Rails (there was no pure Ruby data, which makes sense for BuiltWith's purpose):

Framework Usage


Essentially, PHP pops up 30 more times in websites than Ruby but PHP usage is essentially stable while Ruby's is on the rise; so it's worth another look at whether the frequency of vulnerabilities is related to usage or adoption.  Here's vulnerabilities found over time and basic trends in each language:

CVE Counts per Year
Language2012201320142015 ForecastLinear Regress
PHP171127150128.3333333-10.5
Ruby33752739-3
* note Google Slide FORECAST uses a slightly different model than SLOPE, so calling that out.

Contrast the vulnerabilities trend in Ruby -- a decreasing rate of occurrence -- with the adoption rate of Ruby according to BuiltWith -- a ~33% annual growth.  Of course, there's many reasons for finding vulnerabilities, like a large adoption may mean more hackers want to target a larger victim population, or popularity could drive up the amount of bugs in an effort to meet many (possibly insecure) feature requests.  To play Devil's Advocate, PHP does indeed have a decreasing rate of vulnerabilities, but it isn't being adopted more than Ruby. So does a 20-fold difference in vulnerabilities between PHP and Ruby get explained away by their sheer online presence?

Part 2: Writing Bugs


Instances and rates of vulnerabilities don't capture the severity of each vulnerability, or the ease of a hacker exploiting the software, or the laziness of programmers.  (CVSS does focus our attention, though.)  Also, cybersecurity news only gives us superficial  recommendations like "security patch available, update now!" instead of digging deeper into buggy software.  At most, we may read that a "XSS" or "SQL injection" vulnerability was found -- as if those phrases invoke some kind of meaning into our decision to use one software over another.  I decided to dive deeper into one of these XSS & SQL injection Wordpress bugs, similar to one that caused the US Cert issued last week, and found some disturbing practices in PHP programming and some ignored best practices.

One recent vulnerability in Wordpress came down to programmers being too lazy to scrub data input.  Something as seemingly benign as a Wordpress forum was exploited by submitting HTML into the comment field; and because Javascript can be embedded into any valid HTML data stream, the PHP server parsed and rendered back to the web browser client whatever had been submitted as a comment.  The marvel of this hack is that Wordpress comment moderators are often Wordpress administrators who have logged in with unlimited access to the site, so malicious Javascript embedded in a comment field on their web browsers would gain unauthorized, elevated privileges to execute against the entire website.  The studious guy who found this vulnerability made a horrific demo of Javascript uploading content to the PHP server without the moderator's/administrator's knowledge.  That demo leverages cross-site scripting (XSS) to upload the content but it could have injected malicious SQL into Wordpress.

I've respected the programmer's motto of being lazy and keeping things simple but after seeing this hack I wonder: have we become too lazy?  Apparently the Wordpress comment functionality above wasn't unknown to developers.  Wordpress designed the comment form this way as a feature -- Wordpress users wanted to "texturize" comments with options like italicized fonts, embedded hyperlinks, etc. so Wordpress developers enabled HTML parsing of comments by the PHP engine.  They were actually doing a kind of data filtering of the comment field but did not thoroughly sanitize it!

In the first chapter of his concise work PHP Security, Shiflett's rationale for bringing up thorough data scrubbing at the very start of his book harkens back to the CVE reports I've cited before:

The vast majority of security vulnerabilities in popular PHP applications can be traced to a failure to filter input. (pg. 21)

A bit of Googling on problems with PHP programmers filtering data returned some disturbing practices.  For example, PHP had come up with a global requirement for quoted data to be escaped to prevent SQL injections but Shiflett notes that in reality this caused complications that encouraged programmers to fall back to merely stripping data of quotes or slashes instead of checking for valid data.  When this global quoting requirement troubled one PHP programmer, not a single suggestion included either checking for valid data or using best practices in filtering data.  Shiflett suggests using standard data sanitation functions in PHP, including htmlentities() for interacting with front-end data (and its exhaustive ENT_QUOTES parameter), mysql_real_escape_string() for back-end data, etc.  Why were these functions not included in answers for the PHP programmer having issues with quoted data?

To go back to the above case of the Wordpress vulnerability, what constitutes "thorough" data sanitization?  In the most strict sense, the Wordpress fix required not allowing HTML input from the user that hadn't been predefined. To keep that feature in tact, Wordpress developers had done data filtering that only allowed a subset of HTML to be valid input, but that kind of sanitation was undermined by forgetting another data sanitization process: not checking for valid data size.  The hack demo'ed above, after all, leveraged a very lengthy comment field.  So the vulnerability was incomplete or superficial data sanitation.  If I extrapolate these mishaps to web application security for Ruby, then there isn't a magic pill.  Ruby or Rails might have more programmatic means for data sanitation but the same fundamental process applies.  (I'll leave Ruby data sanitization for a follow-up blog.)  Generally speaking then, I don't buy the excuse that vulnerabilities are merely a lack of forethought.  It gets back to how we program.

Preventing Bugs


One of my coworkers boiled this entire blog down to "bad coding" but I think that is laziness itself speaking on our behalves.  The takeaway I got from writing about this XSS and SQL injection bug is the latent, perennial problem of feature requests over best practices.  Allowing texturized comments was valued more than allowing thorough data sanitization.  In the PHP / Wordpress examples above, data sanitation was needed for:

  1. String fields of maximum length
  2. String data input of only predefined HTML tags

More generally, the development lifecycle would include systematic means for exhausting the scrubbing of data input and output.

Programmers have a means for various testing, and can include a good kind of testing in-flight with unit tests.  For every input or output operation, we should iterate over data that abuses the interface to determine whether more data sanitation is needed.  This leads to one characteristic of Ruby and PHP where the languages differ.  Ruby includes a standard, unit test out-of-the-box whereas PHP programmers must choose and install one of many frameworks to start doing them.

These lessons learned aren't news.  Even good Ruby web applications will need to be written with the idea of systematically preventing bugs and weaknesses by exhaustively testing data sanitation.  MITRE's common weaknesses database is blatantly sarcastic about their best practices being ignored by programmers, as attested by their reports.  (See CWE-79, and CWE-434).  To be fair, MITRE's recommendations can be harsh, like not trusting any input, even a PNG sourced by HTML, so some of the lessons boil down to classic security debates about functionality.  And to be honest, I haven't typically tested for unexpected data, have only done the minimum in leveraging programmatic data filtering, and I've never included unit tests in my own programs.  These are all recommendations I make knowing that being lazy isn't an excuse.  What is left unknown is whether PHP programmers are more lazy than Ruby programmers.  :)