SIGCOMM 2010: Day 3

Network IDS

NetFence: Preventing Internet Denial of Service from Inside Out

  • DDoS is projected to be the biggest problem facing the internet in the next 12 months, and it is difficult to combat, since it conflicts with the openness and robustness internet design principles.
  • Previously, people have looked at the receivers of the DDoS (Denial of Edge Service). Usually using network filters or network capabilities.
  • But with a large enough bonnet, bots can collude to send packet floods which impair network services.
  • Challenge is to design a network architecture that combats both kinds of attack.
  • Solution: NetFence. Gives the network control over its resource allocation to combat denial of network services (DoNS). Also hierarchical and coupled with network capabilities.
  • Hierarchical congestion policing slows down flooding senders, and is robust to both compromised routers and hosts. Uses symmetric key cryptography, and each packet carries a secure token (based on Passport from NSDI 2008).
  • Secure congestion policing feedback are like network capabilities. Capabilities are returned if the receiver authorizes the traffic as “desired”.
  • Two types of packet: request and regular. Packet has five fields: mode (nop/monitor), link ID, action (up or down), timestamp and MAC (authentication).
  • First a sender sends a request packet. The mode field gets stamped by the access router as nop, and the MAC is calculated based on a hash of the original fields. The action gets set to down (deprioritize) the Link ID is stored and the mode is set to monitor, if an attack is deemed to be underway (i.e. congestion is encountered). The routers have keys distributed using Diffie-Hellman over BGP.
  • Policing is done at the access router, which looks at the packet sent back from the receiver (mode, action, etc.), and configures a leaky bucket as necessary.
  • Congestion policing loop uses AIMD at the access router to vary the sender’s bucket capacity.
  • A policing cycle is started based on a load- or loss-based detection procedure in the bottleneck router. RED is used to signal congestion within a cycle.
  • Works because: (i) secret keys used by routers to do feedback, (ii) periodic AIMD used to achieve fairness/efficiency, and (iii) congestion feedback acts as capabilities to prevent unbounded traffic.
  • Provable fairness shown in the paper. i.e. Each good user achieves a proportion of the network capacity that is equal to one over the total number of senders. Denial of service becomes “predictable delay of service”.
  • Many possible attacks against a system like this. Discussed in the paper, but two are discussed here.
  • To deal with floods of request packets, the request packet channel is separate, and there is a per-sender request packet limit, which is policed. There is a priority-based backoff which emulates computational puzzles.
  • To deal with routers hiding backoff feedback, the system treats the absence of an up-feedback as down-feedback.
  • Implemented on Linux using XORP and Click. AES-128 is the MAC function (see Encrypting the Internet). Benchmarked using DeterLab, dual-core 3GHz Xeons with 2GB of RAM.
  • The bottleneck router has 0 processing overhead when there is no attack. Overhead is 492–554 ns/packet (one AES computation) when there is an attack.
  • Shim layer between IP and TCP (or something else), which adds a header overhead between 20–28 bytes.
  • Simulated using NS-2 to evaluate various attacks. Compared to other systems, which put more state in the core.
  • Experiment on a denial of edge services attack. As the number of simulated senders increases, the file transfer time remains constant, unlike Fair Queuing which increase, but TVA+ and StopIt are faster (but less scalable).
  • Experiment on a denial of network services attack. Looked at ratio of average user throughput to average attacker throughput. NetFence achieves fairness.
  • Q. Do you distinguish good and bad users? No, we used AIMD to achieve fairness instead.
  • Q. How can you separate a flash crowd from malicious traffic? We don’t, treating extreme congestion the same way as an attack because it is a failure of end-to-end congestion control.

ASTUTE: Detecting a Different Class of Traffic Anomalies

  • Network management is used to ensure that customer SLAs, security policies, resource availability are maintained. Anomaly detection normally involves building a statistical model of normal traffic and defining an anomaly as a deviation from normal.
  • However, it is hard to obtain a model of “normal” traffic. Look at a time series of packet counts, and usually define a model baseline (tolerance) based on something like EWMA, and anomalies are anything outside that. However, training isn’t guaranteed to be anomaly-free.
  • Aim is to detect anomalies without having to define what is normal. Advantage is a simple tool that doesn’t have to perform training and is hence immune to data poisoning. It is accurate for a well-defined class of traffic anomalies, with theoretical guarantees on the false positive rates. However, its applicability is limited to when traffic characteristics don’t change.
  • Empirical properties: flow independence (although some weak correlation between flows), stationarity (time invariance over the timescales of a typical flow duration), and independence and stationarity => equilibrium.
  • ASTUTE = A Short-Timescale Uncorrelated Traffic Equilibrium. Between two consecutive time-bins, flow volume changes are zero-mean i.i.d.
  • Measure the number of flows, mean volume changes and variance of volume changes between consecutive time bins. Flag an alarm if the ASTUTE Assessment Value (AAV), calculated from these, is greater than some threshold.
  • The threshold controls the false positive rate. Appeal to the central limit theorem, so for a large number of flows, the AAV has a Gaussian distribution. False positive rate is just the area of the bell curve outside the threshold.
  • If ASTUTE is violated, at least one of the model assumptions is violated. For example, stationarity. Long bin sizes (one hour) lead to anomalies flagged when people arrive and leave at the beginning and end of the day (daily bias). Short timescales see no bias at all.
  • Worked with flow traces from Internet2, GEANT2 and the Technicolor corporate network. Compared Kalman and Wavelet filters.
  • Small overlap between anomalies detected by ASTUTE and the other methods. ASTUTE finds different classes of anomalies: tends to be larger numbers of flows with fewer packets than the Kalman and Wavelet approaches.
  • Plotted classified anomalies in each network on a similar graph (#flows vs #packets per flow), and saw that ASTUTE is worse on DoS attacks, but better on prefix outages, link outages and gaps, and port scans.
  • Looked at the ROC curve to see the trade-off between false and true alarms. Kalman would need a much higher false positive rate to detect port scans. But ASTUTE would require a very high false positive rate to detect DoS attacks.
  • Q. Can you not detect large flows because the time windows are so short that they look i.i.d. over those time scales? If it has a small number of flows, it will look independent to ASTUTE. There is an analytical limit to how many flows you need before you can detect it (threshold-squared).
  • Q. Who cares about detecting correlated flows? ASTUTE is not only useful for anomaly detection. But the interesting thing is that it can identify things that the operator would not be aware of, like bugs in misbehaving applications.
  • Q. Do you have the ground truth that the DoS attacks are real DoS attacks? Yes, we have analyzed the data, and there were lots of SYN packets going to a single location, usually from a single IP.
  • Q. Is there a way to classify an anomaly or is it ad hoc? We started with visual inspection, but we developed a tool for this.
  • Q. If your traffic is skewed towards a few flows, does the CLT hold? The CLT and assumption that we have lots of flows is an assumption for normal behavior.

NetShield: Massive Semantics-Based Vulnerability Signature Matching for High-Speed Networks

  • Maintaining network security is a grand challenge. Worms and botnets are widespread.
  • Talk concentrates on signature-based IDS. Normally, there is a database of signatures which is matched against each packet and used to generate alerts. This needs to be accurate and fast.
  • State of the art is regex-based. Used in Cisco IPS, Juniper IDS and Bro (open-source). It can efficiently match multiple signatures simultaneously using an NDFA, and can describe the syntactic conext. But the expressive power is limited, and it cannot describe the semantic context. This leads to inaccuracy.
  • Other state of the art is vulnerability signatures for host-based IDS. It directly describes the semantic context and is very expressive (able to describe the vulnerability exactly). It’s accurate, but slow, using sequential matching and requiring protocol parsing.
  • Vulnerability signature matching requires parsing, matching and combining. Since the protocol grammar is context-sensitive, cannot use a regex, as well as it being practically difficult.
  • Also a regex assumes a single input, so it cannot help with the combining phase.
  • So regex approaches cannot be used to match vulnerability signatures.
  • First challenge: matching thousands of vulnerability signatures simultaneously. Second challenge: parse protocols quickly. Solution achieves 10G throughput with an efficient matching algorithm and a tailored parsing design for high-speed matching.
  • Basically, a vulnerability signature uses a sequence of protocol data units (PDUs) with one predicate per PDU. PDU could be something like the HTTP version or the method. Need numbers and strings, number operators (comparisons) and string operators (equality, regex matching and length)
  • Given n signatures defined on k matching dimensions, a matcher is a two-tuple (field, operation) or a four-tuple for associative array elements. This leads to an n-by-k table. A table representation admits the possibility of matching multiple signatures simultaneously. Table looks like an associative array, with lots of don’t-cares.
  • Worst case time complexity is O((log n)^(k-1)) or O(n^k) space complexity. Based on the Snort and Cisco rulesets, which have selective matchers, the design actually gives O(k) time complexity.
  • Iterative matching algorithm on the columns, based on intersecting relevant rulesets with special treatment for don’t cares.
  • Complexity of merging requires k-1 merging iterations. Worst case merge complexity is O(n) in the worst case, but for real-world russets it will be more like O(1).
  • For high-soeed parsing, compare tree-based and streaming parsers. Streaming parsers can only retain signature related fields. Built an automated parser generator that builds a parsing state machine for parsing the protocol.
  • Implemented in 10kloc of C++ and 3kloc of Python. Evaluated on 26GB traces from Tsinghua University, Northwestern and DARPA. Run on a P4 3.8GHz with 4GB of RAM. For HTTP, 794 vulnerability signatures, and WINRPC 45 vulnerability signatures. Speedup ratio compared to Binpac is around 11x for non-HTTP and 3–4x for HTTP. Maintained throughput of 2.63 (HTTP in the university) to 17.6 (HTTP at Northwestern) Gbps for parsing and matching. Multicore gives a speedup.
  • Tool available online.
  • Q. Can you go into more details about the memory overhead? DFA requires 5.29GB for 973 Snort rules, whereas NetShield requires 2.3MB. The XFA paper showed 863 rules in 1.08MB. NetShield could improve by implementing XFA.
  • Q. Is it possible to do the massive matching using GPUs? Currently, most connections are independent, so yes probably.
  • Q. Do your scalability results not show that you require a clock cycle per bit? We only have to look at the bits in the signature.
  • Q. What are the advantages of your scheme with respect to XFA? Limited accuracy: XFA would make false positives.

Network Architecture and Operations

R3: Resilient Routing Reconfiguration

  • Failures are common, but today’s emerging applications impose a stringent requirement of network reliability. Plus SLA violations may impact an ISP’s revenue. Aim is to recover quickly from a single or multiple overlapping failures.
  • In 500-link network, failure scenarios up to three links exceeds 20 million. So it is difficult to optimize routing to avoid congestion under all possible failure scenarios.
  • Existing approaches focus exclusively on reachability. But these may lead to congestion and unpredictable performance. Some existing approaches consider only a small subset of failures, or optimize routing after failures, but this is too little, too late.
  • R3 require son enumeration of failure scenarios, is provably congestion-free, efficient in terms of storage overhead and flexible to diverse requirements.
  • Represent network as a graph, with link capacities and traffic demands on each link. Output of R3 is a base routing and a protection routing. Protection routing is a fast rerouting defined for every link that might fail.
  • Idea is to transform topology uncertainty to traffic uncertainty. Routing is optimized for he set of traffic demands on the original topology. Consider the amount of load that is shifted to other links when a failure occurs. If the routing is congestion free, rerouted traffic is less than capacity.
  • R3 has two phases. First, offline precomputation which minimizes congestion for original demand plus rerouting virtual demand on the original topology. The protection routing may use routes that later fail. Solve using a linear programming technique.
  • After a link fails, convert the protection routing for that link into a valid routing that doesn’t use any other failed links. After the failure, need to reconfigure the protection routing, which uses the computed rerouting.
  • Offline precomputation and online recompilation is sufficient to get congestion-free routing. Whether it is optimal for more than one link failure is an open problem. The reconfiguration is order independent, which enables distributed recompilation.
  • Some extensions: fixed base routing, trade-off between no-failure and failure protection to bound the no-failure performance, trade-off link utilization and end-to-end delay, prioritized traffic protection, realistic failure scenarios (share risk and maintenance link groups), and traffic variations.
  • Evaluated on two real networks and a synthetic topology. Compared to various rerouting schemes. Added R3 to OSPF and MPLS-ff. Looked for maximum link utilization.
  • For a single failure, R3 achieves near optimal performance. Under multiple failures, it is at least 50% better than other schemes.
  • Implemented for Linux and Linux MPLS. Emulated the Abilene topology on Emulab. 3 physical link failures simulated. Outperforms OSPF+recon by a factor of around 3.
  • Profiled the precomputation time: less than 36 minutes for each topology and less than 17 minutes for non-generated topologies.
  • Storage overhead is < 300KB in the FIB, and < 20MB in the RIB.
  • Q. Have you looked at how to redistribute traffic after a link returns? Have a reconfiguration rule for failure recovery. It will revert back to the last failure scenario, but the ordering may be different (this is provably alright).
  • Q. Have you looked at the overhead of announcements during convergence under churn? No packets will be dropped during this case.
  • Q. How does your algorithm cope with network partition? Studied this in the paper. In this case, we cannot have reachability, so we cannot have congestion-freedom. R3 will ignore the demands that it cannot fulfill.
  • Q. How does your approach compare against oblivious routing schemes (such as Valiant load balance)? These don’t usually handle a large number of failures. Normally big ISPs see larger number of failures than that.
  • Q. How do you evaluate traffic prioritization? Get 10 different priority classes from a US ISP, and show that IP traffic gets sacrifices to protect VPN traffic.

Mercury: Detecting the Performance Impact of Network Upgrades

  • Networks are becoming more complex and diverse. Software and hardware are both becoming more complex. This makes things more sensitive to network glitches or other performance issues. Purpose is to see whether a change makes the network better or worse performing.
  • Normal intuition is that an upgrade will make things better, but complex interactions can lead to unintended consequences. So it is important to monitor the impact of upgrades. This is hard due to the scale and diversity of different devices. So the challenge is to efficiently monitor at scale.
  • Mercury does automated data mining to extract trends, scales across a large number of measurements and flexibly across data sources, and is easy to interpret. Challenge is how to know when an upgrade happens, what their effect on performance is, and to find common factors in who is affected (or it is network-wide).
  • Could drive upgrade detection from the change management system, but since human information is unreliable, instead mine the configuration and workflow logs. Things like OS version and firmware upgrades are easy to track. However, lots of small configuration changes are not related to upgrades (such as customer provisioning). Out-of-the-ordinary changes are ones that are applied to multiple locations in the network, but rarely.
  • Divide event series (SNMP etc.) into equal time-bins to get a time series. Behavior change detection is based on a persistent shift in levels. Recursive rank-based cumulative sum is used on means, medians, standard deviations or distributions.
  • Identifying commonality (of attributes, configurations, etc.) is a machine learning problem (search in a multi-dimensional space). Use the RIPPER rule learner for this.
  • Sometimes aggregation will erroneously amplify rare events. Solution is to time-align each upgrade to each device (as if the upgrade happened at the same time).
  • Evaluted using close interaction with network operators. Used data sets about router configurations and workflow logs, and performance event series: SNMP and syslogs. Collected this data from a tier-1 ISP over 6 months. 988 routers in the study. Categories of router: core, aggregate, access, route reflector and hub.
  • Upgrade detection evaluated for false positives and false negatives. Threshold varied (frequency of change). Tends to see more false positives than false negatives, but these can be filtered.
  • Mercury reduces the number of upgrade-induced change points that the operator must look at by several orders of magnitude, compared to number of syslog entries. It confirmed the earlier operator findings and showed some unknown to the operator.
  • OS upgrades could cause CPU utilization to go down on access routers, but increases in memory utilization on aggregate routers (larger OS image). Varying changes in the number of layer-1 link flaps. More protection switching events.
  • Firmware upgrades could cause less CPU utilization on the central and CPU-facing routers’ CPUs.
  • Protection switching is line-card protection in customer-facing routers. Failover for the access router that customers connect to. Saw a small increase in the frequency of automated PS events. Time alignment was able to show this problem.
  • Q. Have you thought about the inverse problem where your triggers are the alarms of an anomaly detector, and you want to find the root causes? Problem with that is false alarms. With better anomaly detectors, this might become feasible.
  • Q. What is the time horizon of the attribute changes that you consider? We do persistent change detection, so look at daily averages over a history of about 6 months. We are now looking at whether transient things do matter (for the purpose of meeting SLAs, etc.).
  • Q. Do you monitor link capacity in your system? Currently only look at aggregate router statistics, not particular links/interfaces. We are starting to look into that.

California Fault Lines: Understanding the Causes and Impact of Network Failures

  • Most network failures are not catastrophic. But it’s difficult to collect comprehensive failure data. Lightweight techniques are limited, and special-purpose monitoring is expensive.
  • Contributions: a methodology to reconstruct the failure history of a network using only commonly-available data. Basically a time series of layer-3 failure events. Preferably annotated with the cause and impact of the failure. Data source for this is the syslog and the router configuration files in a version control system.
  • But this data is not intended for failure reconstruction. First rebuild the topology from the configuration file, then replay syslog messages. We also have semi-structured data from the maintenance logs.
  • Looked at CENIC network with 200 routers and 5 years of data (California academic network).
  • Limitations: syslog is sent using UDP which leads to message loss. We might see a series of log messages containing a DOWN followed by a DOWN, so just ignore messages until get back on track. Selection bias in the operational announcements.
  • Comprehensiveness: treat the operational announcements as ground truth and see how many of them have corresponding syslog messages. 97% of announcements were confirmed by the syslog.
  • Accuracy: using Skitter project which does frequent traceroutes to confirm that no packets went over down routers.
  • Validated down states using RouteViews (recorded BGP traffic) to track failure events.
  • 60% of failures last less than a minute, which inhibits detection or recovery. Turns out mostly to be flap events.
  • 7000 emails led to 3000 events. 28% of events are failures and 18% of observed failures are explained.
  • Failure causes: hardware, power, external, software, other and configuration. Hardware is the biggest cause of notices, but software is the biggest cause of failures (32% of failures). But almost 80% of software failures were due to scheduled changes.
  • Q. How are those failures distributed on the network? More at the backbone or on the edge? More downtime on the customer links and the high performance links than on the backbone.
  • Q. How do you think what you show shows more about the impact than simply tracking the control plane? It’s hard to know what the actual impact is, since we don’t collect that information. What other sources of information do we need on top of routing information? If we understood link utilization then we could see how links were being strained by these events.
  • Q. Are you saying that software upgrades are a dominant cause of failures? Not dominant, but serious. The UP/DOWN messages are a side-effect of the maintenance activity? Might be interesting to look at this.
  • Q. Do you see many concurrent failures? More details about this in the paper.

Novel Technologies for Data Center Networks

c-Through: Part-Time Optics in Data Centers

  • Comparing optical circuit switching to electrical packet switching. Circuit switching vs. store and forward. Optical can do 320×100G, vs. 16×40G for electrical. But the optical switching time is about 10ms, compared to packet granularity.
  • Despite slow switching time, optical circuit switching is still promising. Full bisection bandwidth at packet granularity may not be necessary.
  • Looked at a hybrid packet/circuit switched architecture. PS for low latency and optical-CS for high capacity transfer. Optical paths are provisioned rack-to-rack.
  • Control plane needs to estimate traffic demand and configure optical circuit based on it. Data plane does traffic demuxing and optimizes circuit utilization (maybe).
  • c-Through is a specific design for this. A centralized controller manages circuit configuration. Applications and switches are not modified, and end hosts are leveraged for traffic management.
  • Enlarge socket buffers for applications to identify which flows are heavy and which are lightweight. This generates a per-rack demand vector. Applications are unmodified, and packets are buffered per-flow to avoid head of line blocking. This estimates traffic demand and pre-batches data to improve optical circuit utilization.
  • Traffic demand vectors aggregated into a traffic matrix. Use Edmonds’ algorithm to compute the optimal configuration (maximum weight matching problem). Then servers are notified. The control traffic overhead could be reduced.
  • Electrical and optical networks isolated using VLANs.
  • Traffic control on hosts, which makes end-hosts tag packets for the two VLANS. accordingly.
  • Testbed with 16 servers, a hybrid network on a 48-port Ethernet switch. Optical switch is emulated using 4G links, whereas electrical network uses 100Mbps links. Optical circuit emulation: optical paths are only available when hosts are notified. There is a 10ms reconfiguration delay.
  • Evaluated TCP performance using dynamic bandwidth, overhead of traffic control and buffering effects. Also application performance (VM migration, MapReduce, MPI-FFT).
  • TCP exploits the dynamic bandwidth quickly. Throughput ramps up within 10ms. Throughput stabilizes within 100ms.
  • MapReduce performance. Since shuffling is independently transferred, it is amenable to batching. Sorted 10GB of random data, which took 800 seconds on an electrical network. With full bisection bandwidth, the performance is 135 seconds. As c-Through varies the buffer size limit, the best performance is 153 seconds, for 100MB buffers, which is close to ideal. As the reconfiguration interval is varied, can do it as infrequently as every 3 seconds, and the performance is 168s.
  • Ran Yahoo Gridmix benchmark, which contains 3 runs of 100 mixed jobs, such as web query, web scan and sorting. Uses 200GB of uncompressed data, and 50GB of compressed data. c-Through comes very close to the full bisection bandwidth network.
  • Q. Surprised by the claim that TCP works fine in this case, considering the multipath issues: would new protocols not be more appropriate? This technique we didn’t see many things blow up.
  • Q. Do you think it could work if the fibre is cut, and how will it affect the network? Current system doesn’t take this into account, but since there is dynamic monitoring, we could detect that and handle it.
  • Q. Won’t you have to reconfigure faster to catch short, bursty flows, and then isn’t there a risk of oscillations? Didn’t see that in our experiments.
  • Q. What is the cost of these optical technologies, and are they practical today? Expensive fixed cost, but the per-port marginal cost is not so high, which makes it competitive. A mature technology that is already on the market.

Helios: A Hybrid Electrical/Optical Switch Architecture for Modular Data Centers

  • Talk is about combining electrical packet switches (ePSs) and optical circuit switches (oCSs) in a data center network. Both cost $500/port. But ePS is limited to about 1G or 10G (maybe 40G or 100G in future), and oCS is rate-free. ePS uses 12W/port and oCS uses 240mW/port. Finally the oCS doesn’t require a transceiver, which costs another watt per port. But the downside of oCS is the 12ms switching time. ePS suited to bursty, uniform traffic, whereas oCS suitable for stable, pair-wise traffic.
  • Switching delay is due to mirrors on motors which it is necessary to reposition to switch the circuit. This simply gives a full crossbar circuit switch which does not decode packets and needs an external scheduler.
  • Wavelength division multiplexing uses one wavelength for a channel. WDM mux and demux are used on the electrical packet switch transceivers.
  • Need stability to be increased, using aggregation. Processes are more likely to communicate than threads; racks are more likely to communicate than servers; data centers are more likely to communicate than pods. Sweet spot is modular data centers.
  • With 64 pods, with 1024 hosts per pod, with a 10% electrical network (10:1 oversubscribed), need $6.3M, 96.5kW and 6656 cables. With a 100% electrical example, it would cost $62.2M, use 950kW and need 65,536 cables. Helios costs $22.1M, uses 157.2kW and needs 14016 cables.
  • Optical switch has a simple software agent, and the intelligence is in the centralized topology manager. Control loop estimates the traffic demand (hard to do), computes the optimal topology for maximum throughput, and then configure the pod and circuit switches.
  • Estimate: will this flow use more bandwidth if we give it more capacity (is it an elephant flow or a mouse flow)? However the results are biased by the current topology. So use the Hedera algorithm (NSDI 2010) which assumes all hosts are connected to an ideal crossbar switch, then compute the max-min fair bandwidth fixpoint.
  • The optimal topology is computed as a max-weight perfect matching on a bipartite graph, using Edmonds’ algorithm.
  • Testbed used two networks: a traditional one and a Helios network. 100% bisection bandwidth is 240Gb/s. Used 26 servers, and various switches including an optical circuit switch.
  • Ran Hadoop on this network, but didn’t get good numbers because the network was massively overprovisioned.
  • Got 190Gb/s peak and 171Gb/s on average on the traditional network, with drops due to hash collisions. The 50Gb/s difference from the full bisection bandwidth is the TCP overhead.
  • Helios got 160Gb/s peak and 43Gb/s average. Due to some quirks of the packet switched routers, such as port debouncing which prevents false positives on ports being up, which led to poor performance on reconfiguration. Turning that off got the average up the 87Gb/s. Turning off EDC got a 142Gb/s average. Remaining overhead is a limitation in the software. Still have 27ms gaps, due to some switching delay.
  • Helios used unidirectional circuits but there are bidirectional circuits as well. Unidirectional doesn’t waste bandwidth on the return path, which leads to a daisy chain topology.
  • First paper to demonstrate WDM in a hybrid electrical/optical network.
  • Q. Have you thought about how the traffic demand estimation technique would work at lower levels (down to within a pod, a rack, a server, a process)? The Hedera demand estimator works on the level of TCP-flows, so we could do that. Would the bias you get become stronger? [Taken offline.]
  • Q. The number of electrical and optical switches you provision is an a priori design decisions, so how would you address changing traffic patterns? The way around that is to build a hybrid electrical/optical switch.
  • Q. Have you thought about application-limited flows, where there is a bottleneck in the application that stops it using the additional bandwidth? Sensitive to the elephant flow classification. The whole pipeline depends on a good classification. Wouldn’t it be better to use OS modification (per c-Through)? Prefer not to modify the host.
  • Q. What would happen, if you didn’t have such short RTTs (such as in an aggregation network), to the end-to-end flows without buffering? It’s not clear that this would do so well (unmodified) between data centers, but the switching technology is well-suited.

Scalable Flow-Based Networking with DIFANE

  • A scalable way to apply fine-grained policies in enterprises.
  • Want to support flexible policies, such as access control rules, customized routing (e.g. Skype calls on a low-latency path) and measurement (e.g. detailed HTTP traffic statistics).
  • Flow-based switches store their rules in a high-speed TCAM, and perform simple actions based on those rules. The flow space has at least five dimensions. Want to specify these in a high-level management system and enforce low-level rules in the TCAM. Want to support large numbers of hosts, switches and policies with limited TCAM space.
  • If you pre-install the rules in the switches, this is simple, but it doesn’t support host mobility and switches don’t have enough memory for all rules.
  • Alternatively (per Ethane, NOX) install the rules on demand, buffering the first packet while the rules are looked up in the controller. The first packet misses the rules, and gives additional switch complexity, and the risk of DoS by sending multiple different packet headers.
  • DIFANE supports host mobility, reduces memory usage and keeps all packets in the data plane.
  • Stage 1: controller proactively generates rules and sends them to some authority switches. The flow space is partitioned between the authority switches.
  • Stage 2: authority switches keep packets in the data plane. When a packet is received, it is routed to the authority switch and sends feedback of the rules to cache. Subsequent packets hit the cache and are forwarded directly. There is no longer a race between updating the cache and forwarding subsequent packets.
  • A small set of coarse-grained wildcard rules is used to give the partition function for authority switches. Not a DHT, since wildcards are used in the rules.
  • A switch’s TCAM has cached rules, authority rules (if the switch is an authority switch) and partition rules (to route to an authority switch). Prefer cached rules and authority rules over partition rules.s
  • Switch prototype built with an OpenFlow switch.
  • Tricky to cache rules when wildcard rules may overlap (with different priorities). Therefore have to generate new rules based on contiguous subregions. Partition based on minimizing the TCAM entries in switches. Use a decision tree base rule partition algorithm to decide where to place the splits in the flow space.
  • Need to handle policy changes at the controller, topology changes at the switches and host mobility.
  • Evaluated prototype by implementing DIFANE in a kernel-level Click-based OpenFlow switch. Traffic generator, switches and controller run on separate 3GHz Xeons.
  • NOX sees a 10ms RTT delay for the first packet, but DIFANE sees a 0.4ms delay.
  • DIFANE can easily be implemented in hardware, whereas NOX requires more software intervention.
  • For peak throughput (one authority switch, single-packet flow), NOX hits an ingress switch bottleneck at 20Kflows/sec with one ingress switch, and then reaches a controller bottleneck with more ingress switches.
  • How many authority switches? Depends on number of rules. Campus network has 30K rules, which is assumed to be 160KB of TCAM memory. This leads to about 3 authority switches. An IPTV network with 5M rules requires 1.6MB of TCAM and would require 100 authority switches.
  • Tension between distributed (switch-based) and centralized (controller-based, easier to manage) operation. DIFANE is a point in between these extremes.
  • Q. How realistic are your assumed TCAM sizes? Already have 160 KB TCAMs, so we would just use more switches.
  • Q. If you have a slow path you can scale much better, so why do you want to keep everything on the fast path? [Taken offline.]
  • Q. Did you experiment with cache replacement policies? Much work done on how to cache rules, so we can just leverage that.
  • Q. What about the importance of dynamic rules that might change frequently, and how can DIFANE handle it? Think that only traffic engineering needs such dynamic rules. DIFANE can get the controller involved to manage these. But the performance gain is not much over OpenFlow in that scenario. Isn’t a benefit of OpenFlow that you can implement e.g. authentication at the application level? Yes, but we can get the controller to push this into the rules.
  • Q. Is there a cost to have all switches be authority switches? Depends on the network and how it is used. Why not make every switch an authority switch? May need more redirection, and hence more stretch. Also the rules will become smaller.
  • Q. Does this make internet traffic more unpredictable? A reasonable comment, but since we know the location of the authority switch, we know the paths that the traffic may take.

Social Networks

An analysis of Social Network-based Sybil defenses

  • Many online services allow attackers to create accounts for free and they can hence manipulate the system.
  • Defense approaches: trusted certification (such as SSN or passport number), or resource challenges (e.g. cryptopuzzles, not hard to solve if you can get cloud resources on demand). Or can use links in the social network to detect Sybils, since we presume that attackers can only create a limited number of links to non-Sybil users. Spawned a lot of research.
  • Unanswered questions: since the schemes use different mechanisms, it is unclear how the schemes are related, or whether there is a common insight across the schemes? This would help us understand the limitations of the defenses.
  • Talk proposes a new methodology for comparing these systems and finds that they all work in a similar manner. It implies that they have a hidden dependence on the network structure, which identifies the limitations of the schemes.
  • The interesting fact is how these schemes identify nodes as Sybils.
  • Schemes take a social network and a single trusted node, and declares Sybils from the perspective of the trusted node. Internally, each node has a Sybil probability, which gives each node a ranking of Sybilness. Can this ranking be used to compare schemes?
  • Compared rankings from each scheme from the same social graph. The ranking is jumbled between the different schemes. All schemes seemed to have a cut-off point where the partitions were (unordered) equalish.
  • The cut-off point comes at the boundary of the local community. So all schemes are effectively detecting communities. Nodes in the local community are ranked higher, but the ranking within and outwith the community are in no particular order. Can we then leverage the work on community detection to design new approaches?
  • Bad news: this depends on the graph having monolithic community structure, and the characteristics of the community around the trusted node.
  • Does this make certain network structures more vulnerable? Does having more communities make it harder to identify communities? Evaluated this on various real-world social networks. Simulated a Sybil attack by consistently adding Sybils (5% attack links and 25% Sybil nodes). Accuracy measured using ranking, i.e. the probability that Sybils will be ranked lower than non-Sybils. Compared amount of community structure (modularity) to the accuracy. Modularity seems to be negatively correlated with accuracy.
  • How can the attacker use this intuition? Can he do better than just choosing random links? For example, by placing links closer to the trusted node. Then the attacker could blend in to the community of the trusted node. Experiment ranks the nodes and gives the attacker to give the ability to place links randomly among the top N nodes. Smaller N implies an attacker with more control. Graph shows an attacker with more control will reduce the accuracy of the algorithms.
  • Moving forward: could be useful for whitelisting nodes, and could potentially incorporate information from more layers to make the decision about who is a Sybil.
  • Q. Have you evaluated where the number of Sybil nodes far exceeds the number of attack links? The results hold in those settings as well.
  • Q. Attacks are launched from compromised and fake accounts, so how do you deal with this? This violates the basic assumption that the attacker has few real links, so none of these schemes will work.
  • Q. What if the Sybils form multiple communities? No matter the Sybil topology, as long as the number of attack links is small, none of these schemes will work.

The Little Engine(s) that could: Scaling Online Social Networks.

  • Systems should be scalable, but it can be hard to implement and is not necessary at the start of an online service. Of course, this can lead to a success disaster. The cloud gives hardware scalability, but no automatic application scalability.
  • Frontend, stateless components are easy to make transparently scalable, but the data source is a bottleneck.
  • Obvious solution is full replication of the DB, but the state doesn’t decrease with the number of servers. However it maintains data locality.
  • Next most likely solution is horizontal partitioning/sharding, but the splits are disjoint, which is bad news for OSNs. The shards cannot be disjoint, because OSNs involve queries across social links, or data dissemination across social links. Presumably want to colocate all of your friends on the same server.
  • Relational databases don’t perform well under horizontal partitioning, and are expensive, so people use DHTs. These perform better but there is no SQL, less abstraction, and they suffer under high traffic (incest, multi-get hole, jitter). Also a DHT gives random partitioning, which means many servers will be hit with a particular update, and there is a high replication overhead.
  • Can leverage underlying social structure to make the partition. The SPAR (Social Partitioning And Replication) algorithm does this.
  • Algorithm has to be online (due to system and social network dynamics), fast and simple (using local information, a hill-climbing heuristic and back-pressure load balancing), stable (no cascades) and effective (approximates an NP-hard problem, minimize replicas while also maintaining a level of redundancy).
  • Evaluated on real OSN data (Twitter/Orkut/Facebook). Looked at various algorithms, including random partitioning, MO and METIS.
  • SPAR has a lower replication overhead than the other algorithms, with only 22% overhead over the replication constraint.
  • Three-tier application: front end and application logic are stateless on top, with SPAR middleware in the application logic and the data store (to intercept messages). The SPAR controller, partition manager and directory service coordinate the whole system. To applications, SPAR is totally transparent, implemented on top of MySQL and Cassandra, but could be implemented using other things.
  • Evaluated using a non-distributed Twitter clone (StatusNet) and real Twitter data, and saw if it could scale up across 16 commodity desktop machines. The 99th percentile latency for MySQL with full replication was 16 requests per second, whereas SPAR+MySQL does 2500 requests per second. Vanilla Cassandra does 200 req/s, whereas SPAR+Cassandra does 800 req/s.
  • Q. Can you replicate e.g. Facebook pictures based on the groups of friends? The rule is applied when processing the query itself, though some redundant data would be stored.
  • Q. Have you looked at incorporating more dynamic interaction behaviors in the partitioning algorithms? We have considered adding weights.
  • Q. Any thoughts on Diaspora? Only know what I read in the news and that it’s fully distributed, so don’t think there will be such a thing as a global data store.
  • Q. []? The more clustered you are, the less replication you will need. The results are consistent for large data sets.
  • Q. Would the replication overhead for Orkut not be higher? 12 or 16.
  • Q. Where is the notion of load per server? Would this not allocate servers that have absolutely no work to do? Details in paper.
  • Q. Are there not better designs than a read fan-out? Arguably.

Crowdsourcing Service-Level Network Event Detection

  • Want to identify problems that affect end-to-end performance. Do this in an online way with reliable detection and isolation.
  • Idea is to do monitoring at the edge systems and detect drops in performance.
  • Need a system that is scalable, and has localization in time and space. Scalability from passive monitoring, and fully distributed detection. Also privacy, reliability from uncontrolled hosts and wide adoption (incentive to install).
  • Approach: passively monitor local performance information (signals), and detect drops in performance. Then attempt to get group corroboration from other hosts. A likelihood ration distinguishes network effects from coincidence. Store the data distributedly, and give the operator a tap to get that data out.
  • Evaluated the approach using peer-to-peer applications (a natural fit). This gets us an edge trace. The dataset is from a plugin called Ono, which has been installed by 1 million BitTorrent users worldwide.
  • Case study on the BT Yahoo! network, which has information about confirmed events on its web interface. Gives the dates and times of the issue arising and having been fixed.
  • BitTorrent peers monitor many performance signals, both general and protocol specific (like Torrent availability). The individual signals are noisy, having uncontrolled duration and having a wide range of values. Use some moving-average smoothing to make this easier to interpret.
  • To do group corroboration, why might they occur at the same time? Could be service-specific problems (e.g. lack of a seeder), coincidence (noisy local detection), or a genuine network problem. Coincidence becomes very small with a large number of users. Can tune a likelihood ratio knob to make this more or less sensitive.
  • Evaluated in the wide-area. Don’t know the false positive or false negative rates, because ISPs wouldn’t provide information about when their network went down. Therefore use public information from BT Yahoo!, and do some work under NDA.
  • In one month of BT Yahoo! data, detected 181 events and 54 occur during confirmed events. There were 14 other reported problems. Remaining are not necessarily false positives.
  • Worked with a North American ISP under NDA. Detected 50% of events in regions with fewer than 10k subscribers.
  • Evaluated sensitivity to the likelihood ratio, detected problems 2% of the time for small moving average deviations and 0.75% of the time for larger deviations.
  • Deployed as the Network Early Warning System in 48k BitTorrent clients.
  • Q. This seems like the right approach since performance monitoring should be done at the application layer.
  • Q. Do you think that IP prefix or geolocation information would be useful for grouping? Depends on whether groupings are useful to help with the problem. Using IP prefix already.
  • Q. How are your techniques different from the earlier talks on anomaly detection? This is at the user, so the information that comes back is more useful. Why are you using moving averages compared to something more sophisticated? Wanted to implement it simply and get it incorporated in a BitTorrent client. Many schemes assume a long continuous stream of data.
  • Q. Once you have detected the events, what do you do with them? The idea is for operators to go and fetch this information. But there is a root cause analysis problem here, which is important future work in this area.

Leave a Reply


cialis rx coupon 40 mg lasix erythromycin dental uses price viagra singapore generico do viagra 2012 lisinopril 20 mg bid lexapro 20mg pills 40mg cialis online synthroid prescriptions prescription cialis uk discount generic 150 viagra tetracycline 50 mg levitra 10mg use kamagra soft 50mg generic propecia difference generic cialis welfil 20 viagra pill splitting buy levitra 100mg pill prednisone 475 buy levitra 40 mg lisinopril versus ramipril lasix 40 mg nedir usual bactrim dosage xenical user feedback nexium drug mechanism propecia 1 mg effectiveness prescription female viagra clomid 50mg bula viagra bottle price 50mg cialis generic viagra canada francais viagra cost locally paxil versus wellbutrin 40mg nexium bid wellbutrin sr cheap generic levitra 20mg viagra canada. price zovirax suspension infantil prednisone 5mg prescribed wellbutrin in india price lexapro celexa prednisone uses allergies dangers viagra use cialis 50 tablet egyptian costco cialis cost cheap viagra drugs levitra user reports valtrex tablets price nexium sales in 2011 mg propecia augmentin 1000 mg duo generic augmentin dosage lexapro 10 mg weight the buy levitra prednisone uses diabetes diflucan 200 mg dogs buy viagra mexaco liquid viagra pills cost of doxycycline buy 1000 mg zithromax buspar dry skin augmentin causes gas hydrochlorothiazide getting high viagra sale review buying viagra goa zovirax buy uk viagra genuine cheap pharmacy levitra strattera 25 mg cost generic periactin pills tetracycline antibiotics purchase cialis generic wiki prednisone antitumor drug kesan clomid 50 mg buspar paxil interaction buy diflucan 150mg medlineplus lexapro 250 mg zithromax paypal order cialis sample drug like celebrex vegetal viagra 200 mg lisinopril good drug xenical 150 mg african viagra sales kamagra austria prednisone drug facts buspar and seroquel cialis generica 40mg generic viagra medication generic form levitra buy 2.5mg levitra kamagra hoeveel mg buy kamagra bangkok quick delivery kamagra viagra prices sweden wellbutrin no prescription brown tablet 20 viagra tabletat nolvadex.d buying safe viagra zithromax and montelukast celebrex information canada bactrim uk pharmacy posologia lasix 25 mg tetracycline antibiotics drugs wellbutrin xr kaufen generic wellbutrin 150 xl buspar prozac combination doxycycline preventing nausea uk viagra fast augmentin 500 mg pret viagra price shopper what mg viagra us generic cialis usage levitra lexapro nausea morning synthroid cause hyperthyroidism zovirax suspension gsk medicaments cialis 5mg bactrim antibiotic usage pharmacy india viagra viagra buy safe nexium 10 mg price augmentin aukaverkanir cipro 500mg review nolvadex tren use periostat doxycycline 20mg uk supplier cialis discount viagra professional indocin versus celebrex metformin drug uses zithromax drug information buy cialis re nervousness and viagra taking 2 10mg levitra zoloft canada zoloft tablete 50 mg walmart pharmacy xenical valtrex price philippines enterococcus species doxycycline xenical online au costo di zoloft brown viagra pill doxycycline 300 mg acne buspar medication wiki zoloft 100 mg withdrawal getting off lasix zithromax 250 mg uk viagra masticable magnus 50 mg clomid failed herbal viagra 6800 mg canadian costco cialis levitra generic canada xenical user black cialis sukhumvit cialis cost greece xenical rx get off synthroid bactrim induced lupus propecia cost pill kamagra leicester.uk ebay propecia uk prednisone for sinuses lexapro en generico doxycycline uk costs buying generic valtrex nexium 40 mg costs online legal cialis qualitest lisinopril 10mg can 5mg cialis nexium tablete nuspojave cialis brand online cialis generico soft multi focus lasix viagra au online zithromax prescription cost indian generic viagra cialis 20mg teilen lexapro usa levitra 20mg 8 st cialis 10mg price kamagra oberhausen pillole come viagra 4 nexium used free viagra 150 mg edmonton cialis prescription 160 mg lasix drug lexapro used lupus pregnancy prednisone 99cent generic cialis zovirax suspension mucositis augmentin 500mg price cialis 20 mg headache bactrim per discus prednisone mood disorder 20 mg daily cialis levitra thailand kaufen cytotec 200mg sterilet levitra generic viagra viagra discount voucher cialis medicare cost prednisone 5mg arthritis wellbutrin 150 mg enough acheter viagra generic augmentin 250 mg bijsluiter abra generic viagra phenergan plus codeine lexapro 5 mg reviews buspar kaufen cheap viagra europe budget viagra celebrex 200 mg online lexapro 10 mg enough viagra pfizer harga clomid shipped overnight cialis 100 mg lilly cheapest viagra 800mg drug category strattera lasix drug schedule cialis usp prices viagra costco priligy online review prednisone 20 mg watson lisinopril price 10 mg cheap lisinopril hctz viagra generic timeline common buspar dosage viagra prescription dosage viagra 100mg value zithromax pills online synthroid tablets sizes zithromax uses std viagra usa review lowest cost valtrex cialis liquid buy levitra online rezeptfrei diflucan 200 mg cost cialis 40mg generic ranbaxy viagra cheap usage cialis 5mg us viagra pharmacy cialis discount purchase purchase viagra shanghai amoxil 250mg capsules buy legitimate cialis metformin buy online lisinopril mg lexapro and nervousness prescription price propecia clomid target pharmacy doxycycline, 100 mg glucophage xr usos phenergan us esperienze cialis generico viagra online hongkong erythromycin 500mg ingredients cialis 100mg. price lexapro clonazepam together prednisone streuli 20 mg chromium versus metformin cytotec interactions medicamenteuses nexium full price cialis 5 mg doctissimo viagra kaufen preis cialis dosage online valtrex sale buy viagra tesco viagra pricecvs pharmacy clomid 100mg overweight generic lexapro names lexapro and generic zithromax drug indications lasix iv use viagra online paypal phuket cialis cialis 10 mg bestellen augmentin duo cost zithromax 1.0 g tablets generic viagra trusted prednisone 25mg tablets canada levitra safety zoloft 25 mg posologie propranolol online kaufen viagra goes generic 2012 recreational levitra use albuterol generic ventolin taking crushed viagra valtrex price mexico indian viagra equivalent viagra sample generic usos strattera 160 mg celebrex price coupon cialis generic 40 mg. avis kamagra plus 1000 mg zithromax overnight propecia generic us glucophage tablet 500 mg buy zovirax 4.5g zithromax suspension pediatrico bijsluiter celebrex 200 mg xenerx vs cialis lisinopril uses of buspar common dose lowest priced strattera valtrex discounts erythromycin 500mg tablets kamagra pille 100mg nolvadex sale philippines generika cialis 50 mg lasix 80 mg online viagra brand pfizer wellbutrin dose generic 200 online 20 mg cialis clomid tablet pregnancy prednisone buy cheap buy Cialis 60 pills zoloft pill identification synthroid 05 mg who uses lisinopril muse eller viagra thrush diflucan 1 australia online levitra rx cialis cost 10mg propranolol misused zoloft drug recall generic cialis illegal kamagra 20mg buy cialis cheaply buying viagra australia curam versus augmentin buy kamagra polos clomid sinus nexium 20 mg desconto phenergan vs. cheratussin 200 mg kamagra uk levitra costo farmacia buy cialis online 32 valtrex prescription dosage cialis nus efekti cialis splitting 20mg 2011 20 mg propranolol anxiety doxycycline pharmacy 80 mg prednisone daily nexium price medco viagra et musculation generic propecia shedding levitra prices uk viagra causa diabetes kamagra 100mg ervaringen viagra pfizer direkt levitra film tablete uk viagra liquid celebrex 200 mg tabletki cytotec uso gastrico order propranolol order lexapro cheap buy levitra 30 pills buspar or klonopin viagra coupons pfizer doxycycline and tinnutus doxycycline 100mg cats metformin bluefish 850 mg cialis 50mg buy strattera 40 mg alcohol 60 mgs of levitra clomid 50mg treatment zovirax 50 mg creme levitra generic vardenafil zithromax buy generic lexapro facial blushing buy celebrex prescription my rx cialis propecia 4rx online cialis pharmacy cialis strange pill overnight viagra erfahrung cialis 20mg cytotec 200 mcg price cialis online narucivanje 20 mg zoloft cialis costco com buspar dosage sleep zithromax suspension cost doxycycline blue pill prescription for cialis zithromax 500mg 3 days buying viagra 100mg grapefruit juice buspar kosten cialis pfizer erythromycin capsules used levitra dose rxlist buspar kidney stones indianapolis viagra nolvadex buy usa lisinopril twitching muscles buspar works rosiglitazone plus metformin viagra 50mg australia wellbutrin causes hypertension diflucan 100 mg tablets levitra kaufen belgien viagra france price drug info clomid viagra purchase order orlistat canada alli cialis canada sample viagra pricing india cheap viagra edu cheap viagra day lexapro ssri drug nexium 40 mg canada lexapro versus zoloft 4 pills of cialis sedefil online viagra generic wellbutrin ir crushed wellbutrin sr pcos 150 mg clomid uses of cialis cialis 10 mg wiki nexium 40mg spain levitra 10 mg canada lloyds pharmacy phenergan generic lisinopril purchase about viagra 800mg cialis lilly 40mg xenical online australia cipro 500 mg pill olanzapine fluoxetine generic india shipping cialis xenical 120 mg fiyati nexium 40 canada synthroid drug reactions periactin online buspar spanish lexapro muscle stiffness liquid nolvadex sale levitra als schmelztablette buy zoloft zovirax cure thrush cialis prescription cheap erythromycin ec 333 mg easy delivery viagra viagra tablet effects levitra generic 40 mg. chinese clomid tablets 30 mg cialis dose order cialis prescription cialis tabletke douleur sous clomid viagra pussy.com valtrex alcohol use cialis generique 20mg lasix backorder metformin 1000 mg spc 5mg cialis 36 hours clomid in australia xenical tablete nuspojave nexium cause depression usp prednisone msds nolvadex generic cheap walmart cialis price valtrex online prescription generic augmentin dosing phenergan generic equivalent cialis 20g india levitra 20mg instructions bactrim pill id pfizer viagra recall cialis 10mg coupon levitra 5mg bestellen metformin 850 mg tds teva wellbutrin generic bill viagra use cialis 3 capsules 100mg levitra generic wiki erythromycin 250 mg effectiveness hydrochlorothiazide 25 mg tablets kurus dengan xenical lexapro nausea diarrhea doxycycline 20mg generic personaggi illustri cipro lasix 4 mg levitra canada zamowienie cialis 20 mg kullanimi apo metformin price cialis 5 mg dauertherapie nexium tablete cena 25 mg zoloft anxiety zithromax 250mg israel odchudzanie tabletki xenical 25 mg lexapro use viagra drug buspar maximum dosage lexapro 20 mg controlled severe nausea zoloft canada levitra online levitra sale 5mg doxycycline 50 mg bid cheapest clomid uk buspar uk nhs lasix 40 mg tablets 10 mg de viagra online cialis canadian nolvadex 60mg priligy pills priligy costo farmacia walmart valtrex cost synthroid generic image fedex overnight cialis cialis australia.cs diflucan dosage thrush costo pillole viagra augmentin 500mg tab bisoprolol 5mg viagra nuova pillola viagra buspar cognitive effects doxycycline gum disorder cymbalta lexapro together viagra online hyderabad lloyds pharmacy propecia cialis mg 5 prezzo levitra uk cost metformin drug screen componentes lisinopril 20 mg viagra 12.5mg experience metformin pill z 70 buy 40 mg lisinopril erythromycin s aureus herbal viagra plus cialis two 5mg phenergan for sale cialis lowest prices doxycycline generic dosage generic levitra fda faer cialis prices kamagra jel 5mg xenical uk pharmacy lilly cialis 5 mg propranolol amitriptyline together generic lexapro called jovem usando viagra metformin ukpds study lasix prescription information buspar drug classification amoxil 250 mg 5 cc bula propranolol 10mg sandoz generic augmentin generic viagra ciple prescription medicine nexium lisinopril mepha 5mg bactrim suspension 120mg prednisone 30mg day lasix 400mg prednisone 5 mg taper prednisone tablet dosages viagra pills uses order nolvadex 400 mg of wellbutrin boots cialis cost 150mg clomid follicles amoxil 100mg cheap viagra yahoo glucophage diet pill nigeria bactrim price synthroid de 50mg nexium online sale cialis canada .99 buspar and steroids buspar for clenching viagra professional 100 mg metformin drug testing levitra pill shape xenical drug label abuso del viagra zoloft 50 mg filmtabletten buy zithromax pharmacy kamagra gold cheaper xenical pills use effetti pillola viagra generic cialis sweden augmentin na krztusiec buspar prn generic nexium availability viagra pills soho erythromycin from canada metformin generic problems oral thrush prednisone erythromycin 250mg phap Levitra pill online cialis nongeneric bactrim uses diarrhea makan clomid 100mg propecia finasteride uk kamagra plus paypal zithromax dosage pfizer synthroid 0.075mg tab cialis 20 mg etkileri buy priligy 30mg cialis purchase online nexium us launch kgr 100 pill kamagra synthroid 75 mcg tablets non prescription periactin get levitra prescription propecia costs comparison phenergan 12.5 mg tablets clomid research uk buspar sulfa phenergan medicine used viagra 50mg cost discounted canadian viagra viagra 100mg erfahrung order real cialis flagyl cipro nausea bactrim suspension shqip mauritius levitra tablets intravenous bactrim dosage pfizer viagra szabadalom strattera ritalin together cialis 5mg cijena lexapro buspar combination crushing up lexapro viagra australia 100mg viagra a cukrovka 50mg viagra reviews kamagra uk max discount 150mg viagra males using clomid silvio berlusconi viagra viagra 100mg china celebrex muscle spasm cialis 5 mg chronic viagra online register augmentin price india valtrex dosage generic wellbutrin generic a 102 cheap lisinopril prednisone 20mg bula clomid steroids australia cipro uti epidydimititus xenical australia cost erythromycin 250 mg tablets viagra prices nz augmentin streptococcus pneumoniae augmentin for proteus erythromycin cause acne drugs containing hydrochlorothiazide cialis aching muscle kamagra tyggetabletter wellbutrin 300 xl generic cialis 100 mg 10 tablett chinese viagra pills lasix medication australia buspar dosage lexapro 20 mg drug clomid zum muskelaufbau zithromax herx 1000 mg cipro gonnorhea synthroid rx assistance cheap augmentin 875 cytotec uso seguido towards metformin prodrugs clomid pct cost diflucan 200 mg prospect clomid dosage use viagra 25mg boots ordering metformin online clomid 50 mg embarazo augmentin cause gas levitra 20 mg indicazioni uses for cipro 500 excessive prednisone use synthroid mg mcg generic lexapro manufacturers 100 mg soft cialis cialis 20 mg etkisi 100 mg lexapro tetracycline hydrochloride usp doxycycline 150 mg dosage cialis 100mg mexico cialis 5mg efeito propranolol uses hyperthyroidism erythromycin and thrush prednisone edematous buspar vs atarax clomid without prescriptions zithromax 400 mg suspension nexium wholesale price buy cialis florida buspar zombie celebrex 200 mgm doxycycline drug action taking 75 mg wellbutrin strattera price costco lexapro buy generic levitra de 100 mgs buy propecia bangkok buspar double dose kamagra aus niederlande cheap generic zoloft canadian viagra soft doxycycline price comparison 200 mg doxycycline gum levitra 20 canadian viagra netherlands prescription prednisone buying online clomid order paypal clarithromycin versus erythromycin stopping prednisone use cialis cheapsarasotafl street cost viagra what causes synthroid nexium generic prices buspar patient teaching cipro resistance india celebrex drug contraindications medicamentos zoloft 50 mg cialis 20mg valor buspar hearing loss doxycycline haemophilus influenzae buspar depersonalization generic medication lasix precio lisinopril 5 mg buspar available generic viagra tablet spray chennai viagra price tijuana generic cialis generic clomid success cialis hepatitus c nexium customer strattera 40 mg street 60mg prednisone clomid online price nexium causes gastroparesis buy strattera american generic xenical kaufen viagra australian supplier 20 mg doxycycline hyclate zithromax price 2gram buspar flexeril interactions overdose lisinopril 20 mg buspar focus nexium 10 mg praf teva generic zoloft order zovirax ointment buy cipla clomid buspar hives selegiline plus wellbutrin viagra pills means buspar vikt doxycycline 100mg dogs doxycycline dosis vergeten augmentin tablet sizes sample viagra pills cytotec ghana buy cialis 10 mg durata buspar and vertigo lasix 500mg price us viagra sales doxycycline muscles viagra cobra india cialis tablets use female viagra australia doxycycline tablets crush nolvadex order online telmisartan hydrochlorothiazide price 100mg diflucan zovirax tablets generic valtrex prescription coupon diflucan generic price cost and cialis costo compresse cialis 20mg viagra crush valtrex pills cialis target walmart diflucan hiivatulehdus cipro virus electrical buspar switch viagra from pfizer metformin pill shape cialis online 2.5mg ambien wellbutrin together cytotec tablets pregnancy propecia pharmacy.net zovirax uso ocular apo propranolol 40mg 20 mg to levitra herpes drugs valtrex online viagra perth nolvadex 20 mg kaufen different wellbutrin generics glucophage merck uk prix doxycycline 100mg lichen planus celebrex online cialis uk questionnaire clomid online thuoc nexium 20mg online levitra uk augmentin veterinary use buspar dreams produk nolvadex 300 mg zoloft dosage priligy sustancia activa fibrocartilaginous embolism prednisone lasix sold canada diflucan 150 mg 1 caps metformin eating disorders xenical boleh kurus wellbutrin and tussin buspar anxiety depression glucophage 500 mg ulotka generic propecia erfahrung levitra 20 tablets cheapest viagra 50 mg legal kamagra 100mg cialis price boots genuine lexapro price nexium 40 mg faydalari strattera rusia viagra generic swiss nolvadex stack sustanon 250 viagra pills singapore amoxil price compare priligy purchase online kamagra 100mg schweiz propecia costco sale lexapro overdose 1000mg cialis tablete 5 mg viagra australia information longis cialis onlineapotheke bactrim liquid cost prednisone 20 mg 3 days viagra prescription savings price cialis lilly synthroid causing palpitations propecia 5mg uk viagra tribulus viagra tablet rupees thrush after prednisone hibiscus lisinopril cialis vyvanse together zovirax tab 500 mg strattera 40 mg kaufen zithromax drug allergy levitra walmart drugs celebrex cost 200mg wellbutrin sr pill cialis 5 mg spc doxycycline on backorder xenical with prescription canadian healthcare cialis priligy pills uk usando levitra buy synthroid 1500 mg augmentin wellbutrin medlineplus uk clomid doxycycline syrup price lexapro depression drug online herbal viagra dapoxetine tablets tetracycline und pille viagra lausanne propecia merck online order propecia online 503 xenical in canada 50mg clomid success generic viagra sverige facial flushing celebrex metformin und haarausfall nolvadex forum order erythromycin canadian pharmacy zithromax 500mg 3 pack viagra costs honolulu flexeril celebrex together compare prices levitra viagra 800mg pill generic viagra dubai cialis 40 mg cheap prednisone 10mg identification buying hydrochlorothiazide online mirtazapine plus wellbutrin synthroid getting high canada viagra safe pfizer cipro recall merck propecia online doxycycline 100mg warfarin strattera 10mg cokmudur valtrex mouse traps buy cialis tablets review 2.5 mg cialis levitra belgien kaufen viagra cheap 150 cialis singapore price sales on viagra cialis 2.5 mg discount viagra tablets dose cialis brand prices free cialis prescription prednisone leukemia clomid tussentijdse bloeding price propecia canada diflucan 3 pills zoloft 150 mg emotions trazodone and buspar buspar 5 ht metformin tablets uses viagra competitor costs 5mg lexapro depression buy diflucan tablet viagra canadian sales zithromax one susp cialis 20 mg 100 tablets about cialis online pfizer levitra strattera drug label cialis discount uk doxycycline vergeten lexapro zyprexa together online lexapro pharmacy augmentin uses acne reviews generic lexapro kamagra apotheke online glucophage cause twins lisinopril mucus buy viagra edu doxycycline drug uses of nexium propecia canada fertility cialis generico rischi viagra discount pharmacy 30mg prednisone day nexium 40 mg walmart doxycycline cyprus lexapro 10 mg anxiety female kamagra tablets lasix generic dosage augmentin duo tabletta amoxil 875 mg dosage nolvadex kaufen rezeptfrei australia viagra customs cialis generico delivery viagra cheap 50 mg costo cytotec bogota bayer levitra uk cialis tablets prescription viagra 5mg types Generic Levitra order erythromycin molecular target get viagra phuket cialis discount coupons generic viagra box cialis aurochem india buy generic wellbutrin opus nexium using levitra everyday viagra label humorous dogs zithromax buy priligy mercury drug 36 hour cialis mg zithromax price cebu diabetes viagra prescription viagra retail discount cialis 100 mg kullananlar tussentijdse bloeding clomid prednisone muscle tissue can viagra use diflucan tablet form order 50mg viagra buy amoxil cheap nortriptyline zoloft together lexapro tussin generic zovirax buy prednisone causing headaches intuniv versus strattera buspar vs alcohol hydrochlorothiazide purchase online generic cialis 50 mg carlsbad cialis delivery indication lasix 40 mg abilify plus zyprexa diflucan suspension pregnancy viagra cost australia 2011 cash price nexium viagra kaufen wo propecia in theuk uses of zoloft zovirax wo kaufen prednisone for pertussis propecia versus rogaine diflucan uk price indications for buspar xenical alli drug nolvadex 20mg tablets buy bayers levitra buy viagra 100g valtrex discount program ip generics cialis my pills cialis celebrex 200mg purchase canadian customs viagra cheap kamagra uk 40 mg of propranolol lexapro muscle tightness levitra online pharmacy viagra bloedneus kaboon cialis online viagra online com bactrim versus neosporin zithromax 250mg canada us viagra price sospensione brusca zoloft levitra customer reviews cialis cause afib 50mg viagra fiyati recipes cialis usa viagra 30tablet viagra women uk cumpara cialis online dapoxetine 60 mg canada lightheaded using cialis synthroid cost 2012 zovirax prescription canada augmentin 625 mg bijsluiter viagra vaikutusmekanismi zithromax sp 200mg pemakaian clomid tablet asal usul viagra propecia australia advertisement wholesale c20 viagra cost for bactrim buspar medication information synthroid mgs viagra cost indonesia lisinopril zestril 10 mg buy viagra burnaby nombre viagra generico enterococci tablet bactrim generic cialis webmd levitra half tablet paxil buspar cost nexium 20mg 150 mg de viagra lexapro 5 mg 2012 strattera online discount zoloft 100mg nebenwirkungen canad rx cialis cheap generic cialis cheapest 40 mg. levitra generic viagra 100m 100 mg levitra price effexor buspar anxiety cialis 20mg cheapest phenergan 50 mg pregnancy lisinopril rx5 lasix 20 mg used viagra uk cost levitra 10 mg odt buy cheapest propecia kamagra customs bali cialis 5mg filmtabletten zithromax for canadians doxycycline pbs australia generic cialis penang canada drugs synthroid 250 mg zithromax dosage propecia salem 30 day cost cialis lasix online fedex bactrim without water nexium 20 mg 0800 erythromycin animal use use viagra recreation doxycycline bordetella pertussis buy viagra selfserverx prednisone 10 mg 54 899 viagra prescription labels tetracycline getting pregnant cialis tadalafil generic dtm buspar celebrex online canada 40 mg propranolol anxiety super kamagra 100mg celebrex 200 mg usos xenical diet uk doxycycline et nausee we use doxycycline aarp viagra discount cost doxycycline uk canadian viagra cialis viagra bad pill nolvadex nausea uk levitra prices kegunaan lasix 20 mg rxlist celebrex drug viagra online cheap make viagra costume diflucan 150 mg 3 days erythromycin mercury drug nexium 20 mg 14 comprimidos viagra cialis usa lisinopril 10 mg daily levitra professional canada lloyds pharmacy dapoxetine viagra tabs uk Cialis Tabs USA periactin muscle growth generic clomid canada cost clomid nhs order xenical pills e 20 cialis pill zusammensetzung von cialis pfizer viagra sg prednisone steroid cost viagra plus 400 mg buspar for rage prednisone tablet doses prednisone 80mg using phenergan insomnia doxycycline without insurance canada viagra cialis cialis forum uk price levitra 100 mg erythromycin pharex price kamagra gold tablet cialis mexican pharmacy celebrex canada rx using viagra at 19 cheap rx viagra sublingual viagra 150 mg viagra toppills propecia 1mg boots valtrex uk buy lexapro 10 mg alcohol lexapro klonopin buspar 8 mg of prednisone buy nexium singapore generic levitra strips generic viagra pricemalaysia india viagra name nebenwirkungen cialis 5mg Female Cialis 30 pills zithromax susp neonates cialis price macau buy doxycycline us drug reaction lisinopril prednisone dr house 50 vs 100mg clomid levitra orodispersibile generico buy cheap strattera tabletki poronne cytotec dapoxetine 90 mg review augmentin iv bolus prag levitra kaufen kamagra halv tablett canadian drug viagra viagra pill dimensions strattera 80 mg. price price celebrex walmart lasix use neonates muscle aches, cialis online viagra paypal viagra online vietnam nexium usage dosage nolvadex 20 mg posologie viagra online generico metformin causing itching strattera coupons us glucophage 500 mg tablets zithromax suspension posologia buspar gad buy doxycycline canada cost iv phenergan cialis cause cancer metformin shaped pill cialis 5 cost 1500 mg metformin er viagra biggest customers viagra 200mg pfizer lisinopril mg doses strattera focus crush up cialis zovirax 200 tabletten buy cialis fast propecia bottle cost compounding doxycycline suspension nexium 20 mg packet cibla generic propecia viagra use wikipedia gebrauchsanweisung viagra 100mg 1000 mg cipro xl buspar prezzo kamagra buy line viagra cost 100mg cialis user groups cvs cipro price 5mg cialis urinate viagra tablets women cialis pill identifyer augmentin 675 mg prospect lexapro adjustment time viagra cause blindness viagra uk ireland diflucan thrush treat zoloft 100mg buy cialis 2.5 without prescription nolvadex d generico getting nolvadex australia doxycycline dosage 100mg kwikmed kamagra uk cialis tablets cheap viagra plus dapoxetine cialis information canada viagra usa legal nolvadex online canada cheap cialis buy Purchase Generic Levitra buspar diuretic cheap levitra overnight kamagra 100mg nebenwirkung diflucan is dangerous lexapro for sale buy zoloft generic nolvadex india lexapro drug erowid strattera drugstore.com order generic clomid metformin knuses augmentin cocuklar icin cialis dores musculares viagra causa necrose buy cialis 40mg use cytotec pfizer doxycycline 100mg pimples augmentin fungus cialis prescription men viagra buy paypal clomid thyroid disorders clomid plusieurs ovulation buy viagra online 22 222 buy fish erythromycin celebrex generic prices wellbutrin 50 mg drug class cipro generic lisinopril online cialis 40mg buy kamagra w cukierku lisinopril 20 mg e102 viagra classification uk teva lisinopril tablets cheapest 800mg cialis prednisone 10 mg reviews viagra online real acute sinusitis prednisone 1 tablette viagra kaufen nexium dprice pakistan ampicillin amoxicillin enterococcus prednisone 60 mg prednisolone tetracycline and thrush online prescriptions cialis clomid for sustanon informacion sobre buspar 800 mg viagra uk cialis tablets finland viagra halv tablet pill zoloft 50 mg doxycycline monohydrate tinnitus genericos viagra chile wellbutrin 150 mg bijsluiter augmentin sf suspension nexium 50mg price herpes zovirax tablets hydrochlorothiazide prescription drug celebrex 400mg pfizer viagra werbung viagra stop flushing india sales viagra viagra lichtschalter kaufen buy Cialis Jelly 20 xenical online singapore lasix drug action medicamento doxycycline 100mg viagra tablet weigh cialis 40 mg nederland doxycycline sinusitis dublicate cialis pills order viagra price buy zoloft sertraline lisinopril pill dosage buy zoloft cost celebrex dose australia 100mg viagra review cialis 25 mg usa metformin drug use danemark cialis 8 mg doxycycline prescription information generic pack viagra pill besides viagra lexapro prices walgreens vancouver viagra buy viagra price reduction nexium purple plus viagra 100 usa 3000 mg walmart pharmacy zoloft metformin robitussin zithromax antibiotics online viagra law uk online check cialis levitra 20 mg pill cheap viagra pill viagra patent generic sales cheap kamagra clomid used steroids cialis original use propranolol 80 mg cena order nolvadex online cheap branded viagra nexium plus 40 valtrex monthly cost kaufen levitra deutschland drugs like cialis viagra nebenwirkungen tinnitus 20 mg lexapro depression uk generic cialis synthroid cost walmart viagra cost chennai buspar ativan viagra buy 100 500 mg valtrex daily cialis 200mg tab wieviel mg viagra buy levitra online 7 lexapro versus paroxetine drug classification cipro cipro xl 1000mgibuprofene generic brand celebrex cialis price apoteket pharmacy malaysia viagra vegetal viagra cumpar metformin 500 mg rxlist get viagra now viagra prescriptions tucson xenical orlistat kaufen phenergan tablete dejstvo levitra usual dosage synthroid diet pill zoloft 50 mg weight lexapro pbs price glucophage sivuvaikutukset nexium infusion dosage cialis discount cvs wellbutrin buspar zoloft levitra orodispersible 10mg doxycycline 100mg doziranje lexapro 10mg images cialis preis 5 mg potenzmittel levitra 20mg doxycycline 100mg mrsa levitra normal price synthroid 100 mg abbott celebrex online discount pseudomembranous colitis augmentin lexapro pill shape 10mg generic levitra nexium 10 mg indicaciones prix cialis 5 mg wellbutrin purple tablet generic surplus cialis 400 mg wellbutrin cialis 20mg review free nexium pills 250 mg of cipro buy synthroid 175 mcg precio lexapro 10 mg cialis lutschtabletten generic 40 mg cialis diflucan 150 mgs generic lisinopril price wellbutrin cause hunger lexapro 10 mg wikipedia glucophage 500mg tabs escitalopram 20mg lexapro lexapro generic usa uti 200mg doxycycline prednisone 50 mg taper buspar online pharmacy metformin brustschmerzen zoloft 50 mg emagrece diflucan 200 mg benefits 90 day cialis prescription uso correto viagra is buspar placebo buy zithromax pills buy lisinopril 20mg clomid pills use doxycycline tablets capsules clomid 100mg forums wellbutrin sr discussion prednisone muscle recovery tokobagus viagra xenical otc drug cialis 10 mg original lisinopril tablets india free 20mg cialis cialis 5 mg use legit generic cialis lisinopril versus atenolol prices viagra canada 2013 cialis cramps muscle tinnitus doxycycline cialis reviews 10 mg online propecia prednisone 40 mg tablet valtrex price comparison uk lexapro feline buspar dosage viagra canada women nexium 40 mg colombia zoloft generic forum nexium 100 mg prices levitra dosage alli dietpills 10 mg di viagra viagra pharmacy usa augmentin suspensie de 400 viagra sale qatar viagra tablet description aspen zithromax tabs generic4all dapoxetine buspar valerian root nexium push viagra usa walmart switching propecia generic posologie prednisone sinusite cialis 20 mg wikipedia prednisone 5mg directions use of augmentin 1000 valtrex walmart cost viagra safely online bactrim enterococcus faecalis valtrex cost walgreens how use propecia buspar nicotine metformin dangerous drug prednisone 20mg tablets fertility drugs clomid annunci cialis generico prednisone celebrex together cialis verstopte neus viagra price turkay cost viagra rupees propranolol antagonist drug viagra ve lustral buy viagra express cipro buy nolvadex without prescriptions cialis e reflusso bactrim tablete dejstvo viagra order generic lexapro instructions use viagra tablets hyderabad buy levitra mauritius doxycycline uk buy 250 mg doxycycline acne get cialis commercial cialis 300mg describe lisinopril 10 mg mysterious metformin zovirax cz tablet vistaril and buspar cytotec abortivo uso online daily cialis proteus mirabilis cipro online viagra tablet cialis 5mg timh 1 nexium 40 mg 40mg nolvadex order propecia 1mg synthroid 25 mg bula using glucophage bactrim f sinusitis viagra pfizer us valtrex generic purchase bangkok viagra price augmentin 1 gr tablet buy strattera 18 mg cialis 20mgmos synthroid white pill buspar alcohol buspar dyspepsia hydrochlorothiazide drug bank lexapro 20 mg ocd viagra without arousal kamagra cheapest au daily cialis online nexium cause acne cialis lilly 2.5 pills viagra getup cialis 20mg tarif kamagra 100 online bestellen kamagra overnight usa canadian cialis costs progesterone pills clomid opinion viagra generica thuoc nexium 200mg kamagra birmingham uk buspar hypochondria prednisone zentiva 20 mg mercury drug xenical generic chewable viagra cialis uk boots generic cialis 100 mg prednisone cause cellulite buspar e rivotril 15 mg prednisone dog buspar getting high buspar for sale generic lexapro appetite 500 mg metformin hcl